Model-assisted annotating system and methods for use therewith

ABSTRACT

A model-assisted annotating system is operable to receive a first set of annotation data, corresponding to a broad type of annotation data output. A first training step is performed to train a computer vision model using the first set of annotation data. A second set of annotation data corresponding to the broad type of annotation data output is generated performing an inference function utilizing the computer vision model on medical scans. Additional annotation data further specifies the broad type of annotation data output is received. A second training step is performed to generate an updated computer vision model using set of additional annotation data. A third set of annotation data corresponding to the specified type of annotation data output is generated by performing an updated inference function utilizing the updated computer vision model on medical scans.

CROSS REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No.16/365,787, entitled “MODEL-ASSISTED ANNOTATING SYSTEM”, filed Mar. 27,2019, which claims priority pursuant to 35 U.S.C. § 119(e) to U.S.Provisional Application No. 62/770,334, entitled “LESION TRACKINGSYSTEM”, filed Nov. 21, 2018, both of which are hereby incorporatedherein by reference in their entirety and made part of the present U.S.Utility patent application for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

BACKGROUND Technical Field

This invention relates generally to medical imaging devices andknowledge-based systems used in conjunction with client/server networkarchitectures.

Description of Related Art

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a medical scanprocessing system;

FIG. 2A is a schematic block diagram of a client device in accordancewith various embodiments;

FIG. 2B is a schematic block diagram of one or more subsystems inaccordance with various embodiments;

FIG. 3 is a schematic block diagram of a database storage system inaccordance with various embodiments;

FIG. 4A is schematic block diagram of a medical scan entry in accordancewith various embodiments;

FIG. 4B is a schematic block diagram of abnormality data in accordancewith various embodiments;

FIG. 5A is a schematic block diagram of a user profile entry inaccordance with various embodiments;

FIG. 5B is a schematic block diagram of a medical scan analysis functionentry in accordance with various embodiments;

FIGS. 6A-6B are schematic block diagram of a medical scan diagnosingsystem in accordance with various embodiments;

FIG. 7A is a flowchart representation of an inference step in accordancewith various embodiments;

FIG. 7B is a flowchart representation of a detection step in accordancewith various embodiments;

FIGS. 8A-8F are schematic block diagrams of a medical picture archiveintegration system in accordance with various embodiments;

FIG. 9 is a flowchart representation of a method for execution by amedical picture archive integration system in accordance with variousembodiments;

FIG. 10A is a schematic block diagram of a de-identification system inaccordance with various embodiments;

FIG. 10B is an illustration of an example of anonymizing patientidentifiers in image data of a medical scan in accordance with variousembodiments;

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system in accordance with various embodiments;

FIGS. 12A-12B are schematic block diagrams of a medical scanhierarchical labeling system in accordance with various embodiments;

FIG. 12C is an illustration of an example of a diagnosis prompt decisiontree in accordance with various embodiments;

FIG. 12D is an illustration of an example of a characterization promptdecision tree in accordance with various embodiments;

FIG. 12E is an illustration of an example of a localization promptdecision tree in accordance with various embodiments;

FIGS. 12F-12G are schematic block diagrams of a medical scanhierarchical labeling system in accordance with various embodiments;

FIGS. 13A-13B are schematic block diagrams of a medical scan annotatingsystem in accordance with various embodiments;

FIGS. 13C-13V are graphical illustrations of an example interactiveinterface displayed on a client device in conjunction with variousembodiments;

FIGS. 14A-14C are a schematic block diagrams of a multi-label medicalscan analysis system in accordance with various embodiments;

FIG. 15A illustrates an example embodiment of a model that is beutilized by the multi-label medical scan analysis system;

FIGS. 15B-15D illustrate example embodiments of the multi-label medicalscan analysis system;

FIGS. 16A-16E are schematic block diagrams of a multi-model medical scananalysis system in accordance with various embodiments;

FIG. 17 is a schematic block diagram of a location-based medical scananalysis system in accordance with various embodiments;

FIGS. 18A-18E are schematic block diagrams of a model-assistedannotating system in accordance with various embodiments;

FIG. 19 is a schematic block diagram of a medical scan triaging systemin accordance with various embodiments;

FIG. 20 presents a flowchart illustrating a method for execution by amulti-model medical scan analysis system in accordance with variousembodiments;

FIG. 21 presents a flowchart illustrating a method for execution by amulti-model medical scan analysis system in accordance with variousembodiments;

FIG. 22A presents a flowchart illustrating a method for execution by alocation-based medical scan analysis system in accordance with variousembodiments;

FIG. 22B presents a flowchart illustrating a method for execution by alocal system communicating with a location-based medical scan analysissystem in accordance with various embodiments;

FIG. 23 presents a flowchart illustrating a method for execution by amodel-assisted annotating system in accordance with various embodiments;and

FIG. 24 presents a flowchart illustrating a method for execution by amedical scan triaging system in accordance with various embodiments.

DETAILED DESCRIPTION

The present U.S. Utility patent application is related to U.S. Utilityapplication Ser. No. 15/627,644, entitled “MEDICAL SCAN ASSISTED REVIEWSYSTEM”, filed 20 Jun. 2017, which claims priority pursuant to 35 U.S.C.§ 119(e) to U.S. Provisional Application No. 62/511,150, entitled“MEDICAL SCAN ASSISTED REVIEW SYSTEM AND METHODS”, filed 25 May 2017,both of which are hereby incorporated herein by reference in theirentirety and made part of the present U.S. Utility patent applicationfor all purposes.

FIG. 1 presents a medical scan processing system 100, which can includeone or more medical scan subsystems 101 that communicate bidirectionallywith one or more client devices 120 via a wired and/or wireless network150. The medical scan subsystems 101 can include a medical scan assistedreview system 102, medical scan report labeling system 104, a medicalscan annotator system 106, a medical scan diagnosing system 108, amedical scan interface feature evaluator system 110, a medical scanimage analysis system 112, a medical scan natural language analysissystem 114, and/or a medical scan comparison system 116. Some or all ofthe subsystems 101 can utilize the same processing devices, memorydevices, and/or network interfaces, for example, running on a same setof shared servers connected to network 150. Alternatively or inaddition, some or all of the subsystems 101 be assigned their ownprocessing devices, memory devices, and/or network interfaces, forexample, running separately on different sets of servers connected tonetwork 150. Some or all of the subsystems 101 can interact directlywith each other, for example, where one subsystem's output istransmitted directly as input to another subsystem via network 150.Network 150 can include one or more wireless and/or wired communicationsystems; one or more non-public intranet systems and/or public internetsystems; and/or one or more local area networks (LAN) and/or wide areanetworks (WAN).

The medical scan processing system 100 can further include a databasestorage system 140, which can include one or more servers, one or morememory devices of one or more subsystems 101, and/or one or more othermemory devices connected to network 150. The database storage system 140can store one or more shared databases and/or one or more files storedon one or more memory devices that include database entries as describedherein. The shared databases and/or files can each be utilized by someor all of the subsystems of the medical scan processing system, allowingsome or all of the subsystems and/or client devices to retrieve, edit,add, or delete entries to the one or more databases and/or files.

The one or more client devices 120 can each be associated with one ormore users of one or more subsystems of the medical scan processingsystem. Some or all of the client devices can be associated withhospitals or other medical institutions and/or associated with medicalprofessionals, employees, or other individual users for example, locatedat one or more of the medical institutions. Some of the client devices120 can correspond to one or more administrators of one or moresubsystems of the medical scan processing system, allowingadministrators to manage, supervise, or override functions of one ormore subsystems for which they are responsible.

Some or all of the subsystems 101 of the medical scan processing system100 can include a server that presents a website for operation via abrowser of client devices 120. Alternatively or in addition, each clientdevice can store application data corresponding to some or allsubsystems, for example, a subset of the subsystems that are relevant tothe user in a memory of the client device, and a processor of the clientdevice can display the interactive interface based on instructions inthe interface data stored in memory. For example, the website presentedby a subsystem can operate via the application. Some or all of thewebsites presented can correspond to multiple subsystems, for example,where the multiple subsystems share the server presenting the website.Furthermore, the network 150 can be configured for secure and/orauthenticated communications between the medical scan subsystems 101,the client devices 120 and the database storage system 140 to protectthe data stored in the database storage system and the data communicatedbetween the medical scan subsystems 101, the client devices 120 and thedatabase storage system 140 from unauthorized access.

The medical scan assisted review system 102 can be used to aid medicalprofessionals or other users in diagnosing, triaging, classifying,ranking, and/or otherwise reviewing medical scans by presenting amedical scan for review by a user by transmitting medical scan data of aselected medical scan and/or interface feature data of selectedinterface features of to a client device 120 corresponding to a user ofthe medical scan assisted review system for display via a display deviceof the client device. The medical scan assisted review system 102 cangenerate scan review data for a medical scan based on user input to theinteractive interface displayed by the display device in response toprompts to provide the scan review data, for example, where the promptscorrespond to one or more interface features.

The medical scan assisted review system 102 can be operable to receive,via a network, a medical scan for review. Abnormality annotation datacan be generated by identifying one or more of abnormalities in themedical scan by utilizing a computer vision model that is trained on aplurality of training medical scans. The abnormality annotation data caninclude location data and classification data for each of the pluralityof abnormalities and/or data that facilitates the visualization of theabnormalities in the scan image data. Report data including textdescribing each of the plurality of abnormalities is generated based onthe abnormality data. The visualization and the report data, which cancollectively be displayed annotation data, can be transmitted to aclient device. A display device associated with the client device candisplay the visualization in conjunction with the medical scan via aninteractive interface, and the display device can further display thereport data via the interactive interface.

In various embodiments, longitudinal data, such as one or moreadditional scans of longitudinal data 433 of the medical scan or ofsimilar scans, can be displayed in conjunction with the medical scanautomatically, or in response to the user electing to view longitudinaldata via user input. For example, the medical scan assisted reviewsystem can retrieve a previous scan or a future scan for the patientfrom a patient database or from the medical scan database automaticallyor in response to the user electing to view past patient data. One ormore previous scans can be displayed in one or more correspondingwindows adjacent to the current medical scan. For example, the user canselect a past scan from the longitudinal data for display. Alternativelyor in addition, the user can elect longitudinal parameters such asamount of time elapsed, scan type, electing to select the most recentand/or least recent scan, electing to select a future scan, electing toselect a scan at a date closest to the scan, or other criteria, and themedical scan assisted review system can automatically select a previousscan that compares most favorably to the longitudinal parameters. Theselected additional scan can be displayed in an adjacent windowalongside the current medical scan. In some embodiments, multipleadditional scans will be selected and can be displayed in multipleadjacent windows.

In various embodiments, a first window displaying an image slice 412 ofthe medical scan and an adjacent second window displaying an image sliceof a selected additional scan will display image slices 412 determinedto correspond with the currently displayed slice 412 of the medicalscan. As described with respect to selecting a slice of a selectedsimilar medical scan for display, this can be achieved based onselecting the image slice with a matching slice number, based onautomatically determining the image slice that most closely matches theanatomical region corresponding to the currently displayed slice of thecurrent scan, and/or based on determining the slice in the previous scanwith the most similar view of the abnormality as the currently displayedslice. The user can use a single scroll bar or other single user inputindication to jump to a different image slice, and the multiple windowscan simultaneously display the same numbered image slice, or can scrollor jump by the same number of slices if different slice numbers areinitially displayed. In some embodiments, three or more adjacent windowscorresponding to the medical scan and two or more additional scans aredisplayed, and can all be controlled with the single scroll bar in asimilar fashion.

The medical scan assisted review system 102 can automatically detectprevious states of the identified abnormalities based on the abnormalitydata, such as the abnormality location data. The detected previousstates of the identified abnormality can be circled, highlighted, orotherwise indicated in their corresponding window. The medical scanassisted review system 102 can retrieve classification data for theprevious state of the abnormality by retrieving abnormality annotationdata 442 of the similar abnormality mapped to the previous scan from themedical scan database 342. This data may not be assigned to the previousscan, and the medical scan assisted review system can automaticallydetermine classification or other diagnosis data for the previousmedical scan by utilizing the medical scan image analysis system asdiscussed. Alternatively or in addition, some or all of the abnormalityclassification data 445 or other diagnosis data 440 for the previousscan can be assigned values determined based on the abnormalityclassification data or other diagnosis data determined for the currentscan. Such abnormality classification data 445 or other diagnosis data440 determined for the previous scan can be mapped to the previous scan,and or mapped to the longitudinal data 433, in the database and/ortransmitted to a responsible entity via the network.

The medical assisted review system can automatically generate statechange data such as a change in size, volume, malignancy, or otherchanges to various classifiers of the abnormality. This can be achievedby automatically comparing image data of one or more previous scans andthe current scan and/or by comparing abnormality data of the previousscan to abnormality data of the current scan. In some embodiments, suchmetrics can be calculated by utilizing the medical scan similarityanalysis function, for example, where the output of the medical scansimilarity analysis function such as the similarity score indicatesdistance, error, or other measured discrepancy in one or moreabnormality classifier categories 444 and/or abnormality patterncategories 446. This calculated distance, error, or other measureddiscrepancy in each category can be used to quantify state change data,indicate a new classifier in one or more categories, to determine if acertain category has become more or less severe, or otherwise determinehow the abnormality has changed over time. In various embodiments, thisdata can be displayed in one window, for example, where an increase inabnormality size is indicated by overlaying or highlighting an outlineof the current abnormality over the corresponding image slice of theprevious abnormality, or vice versa. In various embodiments whereseveral past scans are available, such state change data can bedetermined over time, and statistical data showing growth rate changesover time or malignancy changes over time can be generated, for example,indicating if a growth rate is lessening or worsening over time. Imageslices corresponding to multiple past scans can be displayed insequence, for example, where a first scroll bar allows a user to scrollbetween image slice numbers, and a second scroll bar allows a user toscroll between the same image slice over time. In various embodimentsthe abnormality data, heat map data, or other interface features will bedisplayed in conjunction with the image slices of the past image data.

The medical scan report labeling system 104 can be used to automaticallyassign medical codes to medical scans based on user identified keywords,phrases, or other relevant medical condition terms of natural text datain a medical scan report of the medical scan, identified by users of themedical scan report labeling system 104. The medical scan reportlabeling system 104 can be operable to transmit a medical report thatincludes natural language text to a first client device for display.Identified medical condition term data can be received from the firstclient device in response. An alias mapping pair in a medical labelalias database can be identified by determining that a medical conditionterm of the alias mapping pair compares favorably to the identifiedmedical condition term data. A medical code that corresponds to thealias mapping pair and a medical scan that corresponds to the medicalreport can be transmitted to a second client device of an expert userfor display, and accuracy data can be received from the second clientdevice in response. The medical code is mapped to the first medical scanin a medical scan database when the accuracy data indicates that themedical code compares favorably to the medical scan.

The medical scan annotator system 106 can be used to gather annotationsof medical scans based on review of the medical scan image data by usersof the system such as radiologists or other medical professionals.Medical scans that require annotation, for example, that have beentriaged from a hospital or other triaging entity, can be sent tomultiple users selected by the medical scan annotator system 106, andthe annotations received from the multiple medical professionals can beprocessed automatically by a processing system of the medical scanannotator system, allowing the medical scan annotator system toautomatically determine a consensus annotation of each medical scan.Furthermore, the users can be automatically scored by the medical scanannotator system based on how closely their annotation matches to theconsensus annotation or some other truth annotation, for example,corresponding to annotations of the medical scan assigned a truth flag.Users can be assigned automatically to annotate subsequent incomingmedical scans based on their overall scores and/or based on categorizedscores that correspond to an identified category of the incoming medicalscan.

The medical scan annotator system 106 can be operable to select amedical scan for transmission via a network to a first client device anda second client device for display via an interactive interface, andannotation data can be received from the first client device and thesecond client device in response. Annotation similarity data can begenerated by comparing the first annotation data to the secondannotation data, and consensus annotation data can be generated based onthe first annotation data and the second annotation data in response tothe annotation similarity data indicating that the difference betweenthe first annotation data and the second annotation data comparesfavorably to an annotation discrepancy threshold. The consensusannotation data can be mapped to the medical scan in a medical scandatabase.

A medical scan diagnosing system 108 can be used by hospitals, medicalprofessionals, or other medical entities to automatically produceinference data for given medical scans by utilizing computer visiontechniques and/or natural language processing techniques. Thisautomatically generated inference data can be used to generate and/orupdate diagnosis data or other corresponding data of correspondingmedical scan entries in a medical scan database. The medical scandiagnosing system can utilize a medical scan database, user database,and/or a medical scan analysis function database by communicating withthe database storage system 140 via the network 150, and/or can utilizeanother medical scan database, user database, and/or function databasestored in local memory.

The medical scan diagnosing system 108 can be operable to receive amedical scan. Diagnosis data of the medical scan can be generated byperforming a medical scan inference function on the medical scan. Thefirst medical scan can be transmitted to a first client deviceassociated with a user of the medical scan diagnosing system in responseto the diagnosis data indicating that the medical scan corresponds to anon-normal diagnosis. The medical scan can be displayed to the user viaan interactive interface displayed by a display device corresponding tothe first client device. Review data can be received from the firstclient device, where the review data is generated by the first clientdevice in response to a prompt via the interactive interface. Updateddiagnosis data can be generated based on the review data. The updateddiagnosis data can be transmitted to a second client device associatedwith a requesting entity.

A medical scan interface feature evaluating system 110 can be usedevaluate proposed interface features or currently used interfacefeatures of an interactive interface to present medical scans for reviewby medical professionals or other users of one or more subsystems 101.The medical scan interface feature evaluator system 110 can be operableto generate an ordered image-to-prompt mapping by selecting a set ofuser interface features to be displayed with each of an ordered set ofmedical scans. The set of medical scans and the ordered image-to-promptmapping can be transmitted to a set of client devices. A set ofresponses can be generated by each client device in response tosequentially displaying each of the set of medical scans in conjunctionwith a mapped user interface feature indicated in the orderedimage-to-prompt mapping via a user interface. Response score data can begenerated by comparing each response to truth annotation data of thecorresponding medical scan. Interface feature score data correspondingto each user interface feature can be generated based on aggregating theresponse score data, and is used to generate a ranking of the set ofuser interface features.

A medical scan image analysis system 112 can be used to generate and/orperform one or more medical scan image analysis functions by utilizing acomputer vision-based learning algorithm 1350 on a training set ofmedical scans with known annotation data, diagnosis data, labelingand/or medical code data, report data, patient history data, patientrisk factor data, and/or other metadata associated with medical scans.These medical scan image analysis functions can be used to generateinference data for new medical scans that are triaged or otherwiserequire inferred annotation data, diagnosis data, labeling and/ormedical code data, and/or report data. For example, some medical scanimage analysis functions can correspond to medical scan inferencefunctions of the medical scan diagnosing system or other medical scananalysis functions of a medical scan analysis function database. Themedical scan image analysis functions can be used to determine whetheror not a medical scan is normal, to detect the location of anabnormality in one or more slices of a medical scan, and/or tocharacterize a detected abnormality. The medical scan image analysissystem can be used to generate and/or perform computer vision basedmedical scan image analysis functions utilized by other subsystems ofthe medical scan processing system as described herein, aiding medicalprofessionals to diagnose patients and/or to generate further data andmodels to characterize medical scans. The medical scan image analysissystem can include a processing system that includes a processor and amemory that stores executable instructions that, when executed by theprocessing system, facilitate performance of operations.

The medical scan image analysis system 112 can be operable to receive aplurality of medical scans that represent a three-dimensional anatomicalregion and include a plurality of cross-sectional image slices. Aplurality of three-dimensional subregions corresponding to each of theplurality of medical scans can be generated by selecting a proper subsetof the plurality of cross-sectional image slices from each medical scan,and by further selecting a two-dimensional subregion from each propersubset of cross-sectional image slices. A learning algorithm can beperformed on the plurality of three-dimensional subregions to generate aneural network. Inference data corresponding to a new medical scanreceived via the network can be generated by performing an inferencealgorithm on the new medical scan by utilizing the neural network. Aninferred abnormality can be identified in the new medical scan based onthe inference data.

The medical scan natural language analysis system 114 can determine atraining set of medical scans with medical codes determined to be truthdata. Corresponding medical reports and/or other natural language textdata associated with a medical scan can be utilized to train a medicalscan natural language analysis function by generating a medical reportnatural language model. The medical scan natural language analysisfunction can be utilized to generate inference data for incoming medicalreports for other medical scans to automatically determine correspondingmedical codes, which can be mapped to corresponding medical scans.Medical codes assigned to medical scans by utilizing the medical reportnatural language model can be utilized by other subsystems, for example,to train other medical scan analysis functions, to be used as truth datato verify annotations provided via other subsystems, to aid indiagnosis, or otherwise be used by other subsystems as described herein.

A medical scan comparison system 116 can be utilized by one or moresubsystems to identify and/or display similar medical scans, forexample, to perform or determine function parameters for a medical scansimilarity analysis function, to generate or retrieve similar scan data,or otherwise compare medical scan data. The medical scan comparisonsystem 116 can also utilize some or all features of other subsystems asdescribed herein. The medical scan comparison system 116 can be operableto receive a medical scan via a network and can generate similar scandata. The similar scan data can include a subset of medical scans from amedical scan database and can be generated by performing an abnormalitysimilarity function, such as medical scan similarity analysis function,to determine that a set of abnormalities included in the subset ofmedical scans compare favorably to an abnormality identified in themedical scan. At least one cross-sectional image can be selected fromeach medical scan of the subset of medical scans for display on adisplay device associated with a user of the medical scan comparisonsystem in conjunction with the medical scan.

FIG. 2A presents an embodiment of client device 120. Each client device120 can include one or more client processing devices 230, one or moreclient memory devices 240, one or more client input devices 250, one ormore client network interfaces 260 operable to more support one or morecommunication links via the network 150 indirectly and/or directly,and/or one or more client display devices 270, connected via bus 280.Client applications 202, 204, 206, 208, 210, 212, 214, and/or 216correspond to subsystems 102, 104, 106, 108, 110, 112, 114, and/or 116of the medical scan processing system respectfully. Each client device120 can receive the application data from the corresponding subsystemvia network 150 by utilizing network interface 260, for storage in theone or more memory devices 240. In various embodiments, some or allclient devices 120 can include a computing device associated with aradiologist, medical entity, or other user of one or more subsystems asdescribed herein.

The one or more processing devices 230 can display interactive interface275 on the one or more client display devices 270 in accordance with oneor more of the client applications 202, 204, 206, 208, 210, 212, 214,and/or 216, for example, where a different interactive interface 275 isdisplayed for some or all of the client applications in accordance withthe website presented by the corresponding subsystem 102, 104, 106, 108,110, 112, 114 and/or 116. The user can provide input in response to menudata or other prompts presented by the interactive interface via the oneor more client input devices 250, which can include a microphone, mouse,keyboard, touchscreen of display device 270 itself or other touchscreen,and/or other device allowing the user to interact with the interactiveinterface. The one or more processing devices 230 can process the inputdata and/or send raw or processed input data to the correspondingsubsystem, and/or can receive and/or generate new data in response forpresentation via the interactive interface 275 accordingly, by utilizingnetwork interface 260 to communicate bidirectionally with one or moresubsystems and/or databases of the medical scan processing system vianetwork 150.

FIG. 2B presents an embodiment of a subsystem 101, which can be utilizedin conjunction with subsystem 102, 104, 106, 108, 110, 112, 114 and/or116. Each subsystem 101 can include one or more subsystem processingdevices 235, one or more subsystem memory devices 245, and/or one ormore subsystem network interfaces 265, connected via bus 285. Thesubsystem memory devices 245 can store executable instructions that,when executed by the one or more subsystem processing devices 235,facilitate performance of operations by the subsystem 101, as describedfor each subsystem herein.

FIG. 3 presents an embodiment of the database storage system 140.Database storage system 140 can include at least one database processingdevice 330, at least one database memory device 340, and at least onedatabase network interface 360, operable to more support one or morecommunication links via the network 150 indirectly and/or directly, allconnected via bus 380. The database storage system 140 can store one ormore databases the at least one memory 340, which can include a medicalscan database 342 that includes a plurality medical scan entries 352, auser database 344 that includes a plurality of user profile entries 354,a medical scan analysis function database 346 that includes a pluralityof medical scan analysis function entries 356, an interface featuredatabase 348 can include a plurality of interface feature entries 358,and/or other databases that store data generated and/or utilized by thesubsystems 101. Some or all of the databases 342, 344, 346 and/or 348can consist of multiple databases, can be stored relationally ornon-relationally, and can include different types of entries anddifferent mappings than those described herein. A database entry caninclude an entry in a relational table or entry in a non-relationalstructure. Some or all of the data attributes of an entry 352, 354, 356,and/or 358 can refer to data included in the entry itself or that isotherwise mapped to an identifier included in the entry and can beretrieved from, added to, modified, or deleted from the database storagesystem 140 based on a given identifier of the entry. Some or all of thedatabases 342, 344, 346, and/or 348 can instead be stored locally by acorresponding subsystem, for example, if they are utilized by only onesubsystem.

The processing device 330 can facilitate read/write requests receivedfrom subsystems and/or client devices via the network 150 based onread/write permissions for each database stored in the at least onememory device 340. Different subsystems can be assigned differentread/write permissions for each database based on the functions of thesubsystem, and different client devices 120 can be assigned differentread/write permissions for each database. One or more client devices 120can correspond to one or more administrators of one or more of thedatabases stored by the database storage system, and databaseadministrator devices can manage one or more assigned databases,supervise assess and/or efficiency, edit permissions, or otherwiseoversee database processes based on input to the client device viainteractive interface 275.

FIG. 4A presents an embodiment of a medical scan entry 352, stored inmedical scan database 342, included in metadata of a medical scan,and/or otherwise associated with a medical scan. A medical scan caninclude imaging data corresponding to a CT scan, x-ray, MM, PET scan,Ultrasound, EEG, mammogram, or other type of radiological scan ormedical scan taken of an anatomical region of a human body, animal,organism, or object and further can include metadata corresponding tothe imaging data. Some or all of the medical scan entries can beformatted in accordance with a Digital Imaging and Communications inMedicine (DICOM) format or other standardized image format, and some ormore of the fields of the medical scan entry 352 can be included in aDICOM header or other standardized header of the medical scan. Medicalscans can be awaiting review or can have already been reviewed by one ormore users or automatic processes and can include tentative diagnosisdata automatically generated by a subsystem, generated based on userinput, and/or generated from another source. Some medical scans caninclude final, known diagnosis data generated by a subsystem and/orgenerated based on user input, and/or generated from another source, andcan included in training sets used to train processes used by one ormore subsystems such as the medical scan image analysis system 112and/or the medical scan natural language analysis system 114.

Some medical scans can include one or more abnormalities, which can beidentified by a user or can be identified automatically. Abnormalitiescan include nodules, for example malignant nodules identified in a chestCT scan. Abnormalities can also include and/or be characterized by oneor more abnormality pattern categories such as such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, for exampleidentified in a chest x-ray. Abnormalities can also include any otherunknown, malignant or benign feature of a medical scan identified as notnormal. Some scans can contain zero abnormalities, and can be identifiedas normal scans. Some scans identified as normal scans can includeidentified abnormalities that are classified as benign, and include zeroabnormalities classified as either unknown or malignant. Scansidentified as normal scans may include abnormalities that were notdetected by one or more subsystems and/or by an originating entity.Thus, some scans may be improperly identified as normal. Similarly,scans identified to include at least one abnormality may include atleast one abnormality that was improperly detected as an abnormality byone or more subsystems and/or by an originating entity. Thus, some scansmay be improperly identified as containing abnormalities.

Each medical scan entry 352 can be identified by its own medical scanidentifier 353, and can include or otherwise map to medical scan imagedata 410, and metadata such as scan classifier data 420, patient historydata 430, diagnosis data 440, annotation author data 450, confidencescore data 460, display parameter data 470, similar scan data 480,training set data 490, and/or other data relating to the medical scan.Some or all of the data included in a medical scan entry 352 can be usedto aid a user in generating or editing diagnosis data 440, for example,in conjunction with the medical scan assisted review system 102, themedical scan report labeling system 104, and/or the medical scanannotator system 106. Some or all of the data included in a medical scanentry 352 can be used to allow one or more subsystems 101, such asautomated portions of the medical scan report labeling system 104 and/orthe medical scan diagnosing system 108, to automatically generate and/oredit diagnosis data 440 or other data the medical scan. Some or all ofthe data included in a medical scan entry 352 can be used to train someor all medical scan analysis functions of the medical scan analysisfunction database 346 such as one or more medical scan image analysisfunctions, one or more medical scan natural language analysis functions,one or more medical scan similarity analysis functions, one or moremedical report generator functions, and/or one or more medical reportanalysis functions, for example, in conjunction with the medical scanimage analysis system 112, the medical scan natural language analysissystem 114, and/or the medical scan comparison system 116.

The medical scan entries 352 and the associated data as described hereincan also refer to data associated with a medical scan that is not storedby the medical scan database, for example, that is uploaded by a clientdevice for direct transmission to a subsystem, data generated by asubsystem and used as input to another subsystem or transmitted directlyto a client device, data stored by a Picture Archive and CommunicationSystem (PACS) communicating with the medical scan processing system 100,or other data associated with a medical scan that is received and orgenerated without being stored in the medical scan database 342. Forexample, some or all of the structure and data attributes described withrespect to a medical scan entry 352 can also correspond to structureand/or data attribute of data objects or other data generated by and/ortransmitted between subsystems and/or client devices that correspond toa medical scan. Herein, any of the data attributes described withrespect to a medical scan entry 352 can also correspond to dataextracted from a data object generated by a subsystem or client deviceor data otherwise received from a subsystem, client device, or othersource via network 150 that corresponds to a medical scan.

The medical scan image data 410 can include one or more imagescorresponding to a medical scan. The medical scan image data 410 caninclude one or more image slices 412, for example, corresponding to asingle x-ray image, a plurality of cross-sectional, tomographic imagesof a scan such as a CT scan, or any plurality of images taken from thesame or different point at the same or different angles. The medicalscan image data 410 can also indicate an ordering of the one or moreimage slices 412. Herein, a “medical scan” can refer a full scan of anytype represented by medical scan image data 410. Herein, an “imageslice” can refer to one of a plurality of cross-sectional images of themedical scan image data 410, one of a plurality of images taken fromdifferent angles of the medical scan image data 410, and/or the singleimage of the medical scan image data 410 that includes only one image.Furthermore “plurality of image slices” can refer to all of the imagesof the associated medical scan, and refers to only a single image if themedical scan image data 410 includes only one image. Each image slice412 can include a plurality of pixel values 414 mapped to each pixel ofthe image slice. Each pixel value can correspond to a density value,such as a Hounsfield value or other measure of density. Pixel values canalso correspond to a grayscale value, a RGB (Red-Green-Blue) or othercolor value, or other data stored by each pixel of an image slice 412.

Scan classifier data 420 can indicate classifying data of the medicalscan. Scan classifier data can include scan type data 421, for example,indicating the modality of the scan. The scan classifier data canindicate that the scan is a CT scan, x-ray, MM, PET scan, Ultrasound,EEG, mammogram, or other type of scan. Scan classifier data 420 can alsoinclude anatomical region data 422, indicating for example, the scan isa scan of the chest, head, right knee, or other anatomical region. Scanclassifier data can also include originating entity data 423, indicatingthe hospital where the scan was taken and/or a user that uploaded thescan to the system. If the originating entity data corresponds to a userof one or more subsystems 101, the originating entity data can include acorresponding user profile identifier and/or include other data from theuser profile entry 354 of the user. Scan classifier data 420 can includegeographic region data 424, indicating a city, state, and/or countryfrom which the scan originated, for example, based on the user dataretrieved from the user database 344 based on the originating entity.Scan classifier data can also include machine data 425, which caninclude machine identifier data, machine model data, machine calibrationdata, and/or contrast agent data, for example based on imaging machinedata retrieved from the user database 344 based on the originatingentity data 423. The scan classifier data 420 can include scan date data426 indicating when the scan was taken. The scan classifier data 420 caninclude scan priority data 427, which can indicate a priority score,ranking, number in a queue, or other priority data with regard totriaging and/or review. A priority score, ranking, or queue number ofthe scan priority data 427 can be generated by automatically by asubsystem based on the scan priority data 427, based on a severity ofpatient symptoms or other indicators in the risk factor data 432, basedon a priority corresponding to the originating entity, based onpreviously generated diagnosis data 440 for the scan, and/or can beassigned by the originating entity and/or a user of the system.

The scan classifier data 420 can include other classifying data notpictured in FIG. 4A. For example, a set of scans can include medicalscan image data 410 corresponding to different imaging planes. The scanclassifier data can further include imaging plane data indicating one ormore imaging planes corresponding to the image data. For example, theimaging plane data can indicate the scan corresponds to the axial plane,sagittal plane, or coronal plane. A single medical scan entry 352 caninclude medical scan image data 410 corresponding multiple planes, andeach of these planes can be tagged appropriately in the image data. Inother embodiments, medical scan image data 410 corresponding to eachplane can be stored as separate medical scan entries 352, for example,with a common identifier indicating these entries belong to the same setof scans.

Alternatively or in addition, the scan classifier data 420 can includesequencing data. For example, a set of scans can include medical scanimage data 410 corresponding to different sequences. The scan classifierdata can further include sequencing data indicating one or more of aplurality of sequences of the image data corresponds to, for example,indicating whether an MRI scan corresponds to a T2 sequence, a T1sequence, a T1 sequence with contrast, a diffusion sequence, a FLAIRsequence, or other MM sequence. A single medical scan entry 352 caninclude medical scan image data 410 corresponding to multiple sequences,and each of these sequences can be tagged appropriately in the entry. Inother embodiments, medical scan image data 410 corresponding to eachsequence can be stored as separate medical scan entries 352, forexample, with a common identifier indicating these entries belong to thesame set of scans.

Alternatively or in addition, the scan classifier data 420 can includean image quality score. This score can be determined automatically byone or more subsystems 101, and/or can be manually assigned the medicalscan. The image quality score can be based on a resolution of the imagedata 410, where higher resolution image data is assigned a morefavorable image quality score than lower resolution image data. Theimage quality score can be based on whether the image data 410corresponds to digitized image data received directly from thecorresponding imaging machine, or corresponds to a hard copy of theimage data that was later scanned in. In some embodiments, the imagequality score can be based on a detected corruption, and/or detectedexternal factor that determined to negatively affect the quality of theimage data during the capturing of the medical scan and/or subsequent tothe capturing of the medical scan. In some embodiments, the imagequality score can be based on detected noise in the image data, where amedical scan with a higher level of detected noise can receive a lessfavorable image quality score than a medical scan with a lower level ofdetected noise. Medical scans with this determined corruption orexternal factor can receive a less favorable image quality score thanmedical scans with no detected corruption or external factor.

In some embodiments, the image quality score can be based on includemachine data 425. In some embodiments, one or more subsystems canutilize the image quality score to flag medical scans with image qualityscores that fall below an image quality threshold. The image qualitythreshold can be the same or different for different subsystems, medicalscan modalities, and/or anatomical regions. For example, the medicalscan image analysis system can automatically filter training sets basedon selecting only medical scans with image quality scores that comparefavorably to the image quality threshold. As another example, one ormore subsystems can flag a particular imaging machine and/or hospital orother medical entity that have produced at least a threshold numberand/or percentage of medical scan with image quality scores that compareunfavorably to the image quality threshold. As another example, ade-noising algorithm can be automatically utilized to clean the imagedata when the image quality score compares unfavorably to the imagequality threshold. As another example, the medical scan image analysissystem can select a particular medical image analysis function from aset of medical image analysis functions to utilize on a medical scan togenerate inference data for the medical scan. Each of this set ofmedical image analysis function can be trained on different levels ofimage quality, and the selected image analysis function can be selectedbased on the determined image quality score falling within a range ofimage quality scores the image analysis function was trained on and/oris otherwise suitable for.

The patient history data 430 can include patient identifier data 431which can include basic patient information such as name or anidentifier that may be anonymized to protect the confidentiality of thepatient, age, and/or gender. The patient identifier data 431 can alsomap to a patient entry in a separate patient database stored by thedatabase storage system, or stored elsewhere. The patient history datacan include patient risk factor data 432 which can include previousmedical history, family medical history, smoking and/or drug habits,pack years corresponding to tobacco use, environmental exposures,patient symptoms, etc. The patient history data 430 can also includelongitudinal data 433, which can identify one or more additional medicalscans corresponding to the patient, for example, retrieved based onpatient identifier data 431 or otherwise mapped to the patientidentifier data 431. Some or all additional medical scans can beincluded in the medical scan database, and can be identified based ontheir corresponding identifiers medical scan identifiers 353. Some orall additional medical scans can be received from a different source andcan otherwise be identified. Alternatively or in addition, thelongitudinal data can simply include some or all relevant scan entrydata of a medical scan entry 352 corresponding to the one or moreadditional medical scans. The additional medical scans can be the sametype of scan or different types of scans. Some or all of the additionalscans may correspond to past medical scans, and/or some or all of theadditional scans may correspond to future medical scans. Thelongitudinal data 433 can also include data received and/or determinedat a date after the scan such as final biopsy data, or some or all ofthe diagnosis data 440. The patient history data can also include alongitudinal quality score 434, which can be calculated automatically bya subsystem, for example, based on the number of additional medicalscans, based on how many of the additional scans in the file were takenbefore and/or after the scan based on the scan date data 426 of themedical scan and the additional medical scans, based on a date rangecorresponding to the earliest scan and corresponding to the latest scan,based on the scan types data 421 these scans, and/or based on whether ornot a biopsy or other final data is included. As used herein, a “high”longitudinal quality score refers to a scan having more favorablelongitudinal data than that with a “low” longitudinal quality score.

Diagnosis data 440 can include data that indicates an automateddiagnosis, a tentative diagnosis, and/or data that can otherwise be usedto support medical diagnosis, triage, medical evaluation and/or otherreview by a medical professional or other user. The diagnosis data 440of a medical scan can include a binary abnormality identifier 441indicating whether the scan is normal or includes at least oneabnormality. In some embodiments, the binary abnormality identifier 441can be determined by comparing some or all of confidence score data 460to a threshold, can be determined by comparing a probability value to athreshold, and/or can be determined by comparing another continuous ordiscrete value indicating a calculated likelihood that the scan containsone or more abnormalities to a threshold. In some embodiments,non-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the scan contains one or moreabnormalities, can be included in diagnosis data 440 in addition to, orinstead of, binary abnormality identifier 441. One or abnormalities canbe identified by the diagnosis data 440, and each identified abnormalitycan include its own set of abnormality annotation data 442.Alternatively, some or all of the diagnosis data 440 can indicate and/ordescribe multiple abnormalities, and thus will not be presented for eachabnormality in the abnormality annotation data 442. For example, thereport data 449 of the diagnosis data 440 can describe all identifiedabnormalities, and thus a single report can be included in thediagnosis.

FIG. 4B presents an embodiment of the abnormality annotation data 442.The abnormality annotation data 442 for each abnormality can includeabnormality location data 443, which can include an anatomical locationand/or a location specific to pixels, image slices, coordinates or otherlocation information identifying regions of the medical scan itself. Theabnormality annotation data 442 can include abnormality classificationdata 445 which can include binary, quantitative, and/or descriptive dataof the abnormality as a whole, or can correspond to one or moreabnormality classifier categories 444, which can include size, volume,pre-post contrast, doubling time, calcification, components, smoothness,spiculation, lobulation, sphericity, internal structure, texture, orother categories that can classify and/or otherwise characterize anabnormality. Abnormality classifier categories 444 can be assigned abinary value, indicating whether or not such a category is present. Forexample, this binary value can be determined by comparing some or all ofconfidence score data 460 to a threshold, can be determined by comparinga probability value to a threshold, and/or can be determined bycomparing another continuous or discrete value indicating a calculatedlikelihood that a corresponding abnormality classifier category 444 ispresent to a threshold, which can be the same or different threshold foreach abnormality classifier category 444. In some embodiments,abnormality classifier categories 444 can be assigned one or morenon-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the corresponding classifier category 444is present.

The abnormality classifier categories 444 can also include a malignancycategory, and the abnormality classification data 445 can include amalignancy rating such as a Lung-RADS score, a Fleischner score, and/orone or more calculated values that indicate malignancy level, malignancyseverity, and/or probability of malignancy. Alternatively or inaddition, the malignancy category can be assigned a value of “yes”,“no”, or “maybe”. The abnormality classifier categories 444 can alsoinclude abnormality pattern categories 446 such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, and the abnormalityclassification data 445 for each abnormality pattern category 446 canindicate whether or not each of the abnormality patterns is present.

The abnormality classifier categories can correspond to ResponseEvaluation Criteria in Solid Tumors (RECIST) eligibility and/or RECISTevaluation categories. For example, an abnormality classifier category444 corresponding to RECIST eligibility can have correspondingabnormality classification data 445 indicating a binary value “yes” or“no”, and/or can indicate if the abnormality is a “target lesion” and/ora “non-target lesion.” As another example, an abnormality classifiercategory 444 corresponding to a RECIST evaluation category can bedetermined based on longitudinal data 433 and can have correspondingabnormality classification data 445 that includes one of the set ofpossible values “Complete Response”, “Partial Response”, “StableDisease”, or “Progressive Disease.”

The diagnosis data 440 as a whole, and/or the abnormality annotationdata 442 for each abnormality, can include custom codes or datatypesidentifying the binary abnormality identifier 441, abnormality locationdata 443 and/or some or all of the abnormality classification data 445of one or more abnormality classifier categories 444. Alternatively orin addition, some or all of the abnormality annotation data 442 for eachabnormality and/or other diagnosis data 440 can be presented in a DICOMformat or other standardized image annotation format, and/or can beextracted into custom datatypes based on abnormality annotation dataoriginally presented in DICOM format. Alternatively or in addition, thediagnosis data 440 and/or the abnormality annotation data 442 for eachabnormality can be presented as one or more medical codes 447 such asSNOMED codes, Current Procedure Technology (CPT) codes, ICD-9 codes,ICD-10 codes, or other standardized medical codes used to label orotherwise describe medical scans.

Alternatively or in addition, the diagnosis data 440 can include naturallanguage text data 448 annotating or otherwise describing the medicalscan as a whole, and/or the abnormality annotation data 442 can includenatural language text data 448 annotating or otherwise describing eachcorresponding abnormality. In some embodiments, some or all of thediagnosis data 440 is presented only as natural language text data 448.In some embodiments, some or all of the diagnosis data 440 isautomatically generated by one or more subsystems based on the naturallanguage text data 448, for example, without utilizing the medical scanimage data 410, for example, by utilizing one or more medical scannatural language analysis functions trained by the medical scan naturallanguage analysis system 114. Alternatively or in addition, someembodiments, some or all of the natural language text data 448 isgenerated automatically based on other diagnosis data 440 such asabnormality annotation data 442, for example, by utilizing a medicalscan natural language generating function trained by the medical scannatural language analysis system 114.

The diagnosis data can include report data 449 that includes at leastone medical report, which can be formatted to include some or all of themedical codes 447, some or all of the natural language text data 448,other diagnosis data 440, full or cropped images slices formatted basedon the display parameter data 470 and/or links thereto, full or croppedimages slices or other data based on similar scans of the similar scandata 480 and/or links thereto, full or cropped images or other databased on patient history data 430 such as longitudinal data 433 and/orlinks thereto, and/or other data or links to data describing the medicalscan and associated abnormalities. The diagnosis data 440 can alsoinclude finalized diagnosis data corresponding to future scans and/orfuture diagnosis for the patient, for example, biopsy data or otherlongitudinal data 433 determined subsequently after the scan. Themedical report of report data 449 can be formatted based on specifiedformatting parameters such as font, text size, header data, bulleting ornumbering type, margins, file type, preferences for including one ormore full or cropped image slices 412, preferences for including similarmedical scans, preferences for including additional medical scans, orother formatting to list natural language text data and/or image data,for example, based on preferences of a user indicated in the originatingentity data 423 or other responsible user in the corresponding reportformatting data.

Annotation author data 450 can be mapped to the diagnosis data for eachabnormality, and/or mapped to the scan as a whole. This can include oneor more annotation author identifiers 451, which can include one or moreuser profile identifiers of a user of the system, such as an individualmedical professional, medical facility and/or medical entity that usesthe system. Annotation author data 450 can be used to determine theusage data of a user profile entry 354. Annotation author data 450 canalso include one or more medical scan analysis function identifiers 357or other function identifier indicating one or more functions or otherprocesses of a subsystem responsible for automatically generating and/orassisting a user in generating some or all of the diagnosis data, forexample an identifier of a particular type and/or version of a medicalscan image analysis functions that was used by the medical scandiagnosing system 108 used to generate part or all of the diagnosis data440 and/or an interface feature identifier, indicating an one or moreinterface features presented to a user to facilitate entry of and/orreviewing of the diagnosis data 440. The annotation author data can alsosimply indicate, for one or more portions of the diagnosis data 440, ifthis portion was generated by a human or automatically generated by asubsystem of the medical scan processing system.

In some embodiments, if a medical scan was reviewed by multipleentities, multiple, separate diagnosis data entries 440 can be includedin the medical scan entry 352, mapped to each diagnosis author in theannotation author data 450. This allows different versions of diagnosisdata 440 received from multiple entities. For example, annotation authordata of a particular medical scan could indicate that the annotationdata was written by a doctor at medical entity A, and the medical codedata was generated by user Y by utilizing the medical scan reportlabeling system 104, which was confirmed by expert user X. Theannotation author data of another medical scan could indicate that themedical code was generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, andconfirmed by expert user X. The annotation author data of anothermedical scan could indicate that the location and a first malignancyrating were generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, and thata second malignancy rating was entered by user Z. In some embodiments,one of the multiple diagnosis entries can include consensus annotationdata, for example, generated automatically by a subsystem such as themedical scan annotating system 106 based on the multiple diagnosis data440, based on confidence score data 460 of each of the multiplediagnosis data 440, and/or based on performance score data of acorresponding user, a medical scan analysis function, or an interfacefeature, identified in the annotation author data for each correspondingone of the multiple diagnosis data 440.

Confidence score data 460 can be mapped to some or all of the diagnosisdata 440 for each abnormality, and/or for the scan as a whole. This caninclude an overall confidence score for the diagnosis, a confidencescore for the binary indicator of whether or not the scan was normal, aconfidence score for the location a detected abnormality, and/orconfidence scores for some or all of the abnormality classifier data.This may be generated automatically by a subsystem, for example, basedon the annotation author data and corresponding performance score of oneor more identified users and/or subsystem attributes such as interactiveinterface types or medical scan image analysis functions indicated bythe annotation author data. In the case where multiple diagnosis dataentries 440 are included from different sources, confidence score data460 can be computed for each entry and/or an overall confidence score,for example, corresponding to consensus diagnosis data, can be based oncalculated distance or other error and/or discrepancies between theentries, and/or can be weighted on the confidence score data 460 of eachentry. In various embodiments, the confidence score data 460 can includea truth flag 461 indicating the diagnosis data is considered as “known”or “truth”, for example, flagged based on user input, flaggedautomatically based on the author data, and/or flagged automaticallybased on the calculated confidence score of the confidence score dataexceeding a truth threshold. As used herein, a “high” confidence scorerefers to a greater degree or more favorable level of confidence than a“low” confidence score.

Display parameter data 470 can indicate parameters indicating an optimalor preferred display of the medical scan by an interactive interface 275and/or formatted report for each abnormality and/or for the scan as awhole. Some or all of the display parameter data can have separateentries for each abnormality, for example, generated automatically by asubsystem 101 based on the abnormality annotation data 442. Displayparameter data 470 can include interactive interface feature data 471,which can indicate one or more selected interface features associatedwith the display of abnormalities and/or display of the medical scan asa whole, and/or selected interface features associated with userinteraction with a medical scan, for example, based on categorizedinterface feature performance score data and a category associated withthe abnormality and/or with the medical scan itself. The displayparameter data can include a slice subset 472, which can indicate aselected subset of the plurality of image slices that includes a singleimage slice 412 or multiple image slices 412 of the medical scan imagedata 410 for display by a user interface. The display parameter data 470can include slice order data 473 that indicates a selected customordering and/or ranking for the slice subset 472, or for all of theslices 412 of the medical scan. The display parameter data 470 caninclude slice cropping data 474 corresponding to some or all of theslice subset 472, or all of the image slices 412 of the medical scan,and can indicating a selected custom cropped region of each image slice412 for display, or the same selected custom cropped region for theslice subset 472 or for all slices 412. The display parameter data caninclude density window data 475, which can indicate a selected customdensity window for display of the medical scan as a whole, a selectedcustom density window for the slice subset 472, and/or selected customdensity windows for each of the image slices 412 of the slice subset472, and/or for each image slice 412 of the medical scan. The densitywindow data 475 can indicate a selected upper density value cut off anda selected lower density value cut off, and/or can include a selecteddeterministic function to map each density value of a pixel to agrayscale value based on the preferred density window. The interactiveinterface feature data 471, slice subset 472, slice order data 473,slice cropping data 474, and/or the density window data 475 can beselected via user input and/or generated automatically by one or moresubsystems 101, for example, based on the abnormality annotation data442 and/or based on performance score data of different interactiveinterface versions.

Similar scan data 480 can be mapped to each abnormality, or the scan asa whole, and can include similar scan identifier data 481 correspondingto one or more identified similar medical scans, for example,automatically identified by a subsystem 101, for example, by applying asimilar scan identification step of the medical scan image analysissystem 112 and/or applying medical scan similarity analysis function tosome or all of the data stored in the medical scan entry of the medicalscan, and/or to some or all corresponding data of other medical scans inthe medical scan database. The similar scan data 480 can also correspondto medical scans received from another source. The stored similaritydata can be used to present similar cases to users of the system and/orcan be used to train medical scan image analysis functions or medicalscan similarity analysis functions.

Each identified similar medical scan can have its own medical scan entry352 in the medical scan database 342 with its own data, and the similarscan identifier data 481 can include the medical scan identifier 353each similar medical scan. Each identified similar medical scan can be ascan of the same scan type or different scan type than medical scan.

The similar scan data 480 can include a similarity score 482 for eachidentified similar scan, for example, generated based on some or all ofthe data of the medical scan entry 352 for medical scan and based onsome or all of the corresponding data of the medical scan entry 352 forthe identified similar medical scan. For example, the similarity score482 can be generated based on applying a medical scan similarityanalysis function to the medical image scan data of medical scans and402, to some or all of the abnormality annotation data of medical scansand 402, and/or to some or all of the patient history data 430 ofmedical scans and 402 such as risk factor data 432. As used herein, a“high” similarity score refers a higher level of similarity that a “low”similarity score.

The similar scan data 480 can include its own similar scan displayparameter data 483, which can be determined based on some or all of thedisplay parameter data 470 of the identified similar medical scan. Someor all of the similar scan display parameter data 483 can be generatedautomatically by a subsystem, for example, based on the displayparameter data 470 of the identified similar medical scan, based on theabnormality annotation data 442 of the medical scan itself and/or basedon display parameter data 470 of the medical scan itself. Thus, thesimilar scan display parameter data 483 can be the same or differentthan the display parameter data 470 mapped to the identified similarmedical scan and/or can be the same or different than the displayparameter data 470 of the medical scan itself. This can be utilized whendisplaying similar scans to a user via interactive interface 275 and/orcan be utilized when generating report data 449 that includes similarscans, for example, in conjunction with the medical scan assisted reviewsystem 102.

The similar scan data 480 can include similar scan abnormality data 484,which can indicate one of a plurality of abnormalities of the identifiedsimilar medical scan and its corresponding abnormality annotation data442. For example, the similarity scan abnormality data 484 can includean abnormality pair that indicates one of a plurality of abnormalitiesof the medical scan, and indicates one of a plurality of abnormalitiesof the identified similar medical scan, for example, that was identifiedas the similar abnormality.

The similar scan data 480 can include similar scan filter data 485. Thesimilar scan filter data can be generated automatically by a subsystem,and can include a selected ordered or un-ordered subset of allidentified similar scans of the similar scan data 480, and/or a rankingof all identified similar scans. For example, the subset can be selectedand/or some or all identified similar scans can be ranked based on eachsimilarity score 482, and/or based on other factors such as based on alongitudinal quality score 434 of each identified similar medical scan.

The training set data 490 can indicate one or more training sets thatthe medical scan belongs to. For example, the training set data canindicate one or more training set identifiers 491 indicating one or moremedical scan analysis functions that utilized the medical scan in theirtraining set, and/or indicating a particular version identifier 641 ofthe one or more medical scan analysis functions that utilized themedical scan in their training set. The training set data 490 can alsoindicate which portions of the medical scan entry were utilized by thetraining set, for example, based on model parameter data 623 of thecorresponding medical scan analysis functions. For example, the trainingset data 490 can indicate that the medical scan image data 410 wasincluded in the training set utilized to train version X of the chestx-ray medical scan image analysis function, or that the natural languagetext data 448 of this medical scan was used to train version Y of thenatural language analysis function.

FIG. 5A presents an embodiment of a user profile entry 354, stored inuser database 344 or otherwise associated with a user. A user cancorrespond to a user of one or more of the subsystems such as aradiologist, doctor, medical professional, medical report labeler,administrator of one or more subsystems or databases, or other user thatuses one or more subsystems 101. A user can also correspond to a medicalentity such as a hospital, medical clinic, establishment that utilizesmedical scans, establishment that employs one or more of the medicalprofessionals described, an establishment associated with administeringone or more subsystems, or other entity. A user can also correspond to aparticular client device 120 or account that can be accessed one or moremedical professionals or other employees at the same or differentmedical entities. Each user profile entry can have a corresponding userprofile identifier 355.

A user profile entry 354 can include basic user data 510, which caninclude identifying information 511 corresponding to the user such as aname, contact information, account/login/password information,geographic location information such as geographic region data 424,and/or other basic information. Basic user data 510 can includeaffiliation data 512, which can list one or more medical entities orother establishments the user is affiliated with, for example, if theuser corresponds to a single person such as a medical professional, orif the user corresponds to a hospital in a network of hospitals. Theaffiliation data 512 can include one or more corresponding user profileidentifiers 355 and/or basic user data 510 if the correspondingaffiliated medical entity or other establishment has its own entry inthe user database. The user identifier data can include employee data513 listing one or more employees, such as medical professionals withtheir own user profile entries 354, for example, if the user correspondsto a medical entity or supervising medical professional of other medicalprofessional employees, and can list a user profile identifier 355and/or basic user data 510 for each employee. The basic user data 510can also include imaging machine data 514, which can include a list ofmachines affiliated with the user which can include machine identifiers,model information, calibration information, scan type information, orother data corresponding to each machine, for example, corresponding tothe machine data 425. The user profile entry can include client devicedata 515, which can include identifiers for one or more client devicesassociated with the user, for example, allowing subsystems 101 to senddata to a client device 120 corresponding to a selected user based onthe client device data and/or to determine a user that data was receivedby determining the client device from which the data was received.

The user profile entry can include usage data 520 which can includeidentifying information for a plurality of usages by the user inconjunction with using one or more subsystems 101. This can includeconsumption usage data 521, which can include a listing of, or aggregatedata associated with, usages of one or more subsystems by the user, forexample, where the user is utilizing the subsystem as a service. Forexample, the consumption usage data 521 can correspond to each instancewhere diagnosis data was sent to the user for medical scans provided tothe user in conjunction with the medical scan diagnosing system 108and/or the medical scan assisted review system 102. Some or all ofconsumption usage data 521 can include training usage data 522,corresponding to usage in conjunction with a certification program orother user training provided by one or more subsystems. The trainingusage data 522 can correspond to each instance where diagnosis feedbackdata was provided by user for a medical scan with known diagnosis data,but diagnosis feedback data is not utilized by a subsystem to generate,edit, and/or confirm diagnosis data 440 of the medical scan, as it isinstead utilized to train a user and/or determine performance data for auser.

Usage data 520 can include contribution usage data 523, which caninclude a listing of, or aggregate data associated with, usages of oneor more subsystems 101 by the user, for example, where the user isgenerating and/or otherwise providing data and/or feedback that can isutilized by the subsystems, for example, to generate, edit, and/orconfirm diagnosis data 440 and/or to otherwise populate, modify, orconfirm portions of the medical scan database or other subsystem data.For example, the contribution usage data 523 can correspond to diagnosisfeedback data received from user, used to generate, edit, and/or confirmdiagnosis data. The contribution usage data 523 can include interactiveinterface feature data 524 corresponding to the interactive interfacefeatures utilized with respect to the contribution.

The consumption usage data 521 and/or the contribution usage data 523can include medical scan entry 352 whose entries the user utilizedand/or contributed to, can indicate one or more specific attributes of amedical scan entry 352 that a user utilized and/or contributed to,and/or a log of the user input generated by a client device of the userin conjunction with the data usage. The contribution usage data 523 caninclude the diagnosis data that the user may have generated and/orreviewed, for example, indicated by, mapped to, and/or used to generatethe annotation author data 450 of corresponding medical scan entries352. Some usages may correspond to both consumption usage of theconsumption usage data 521 and contribution usage of the contributionusage data 523. The usage data 520 can also indicate one or moresubsystems 101 that correspond to each consumption and/or contribution.

The user profile entry can include performance score data 530. This caninclude one or more performance scores generated based on thecontribution usage data 523 and/or training usage data 522. Theperformance scores can include separate performance scores generated forevery contribution in the contribution usage data 523 and/or trainingusage data 522 and/or generated for every training consumption usagescorresponding to a training program. As used herein, a “high”performance score refers to a more favorable performance or rating thana “low” performance score.

The performance score data can include accuracy score data 531, whichcan be generated automatically by a subsystem for each contribution, forexample, based on comparing diagnosis data received from a user to datato known truth data such as medical scans with a truth flag 461, forexample, retrieved from the corresponding medical scan entry 352 and/orbased on other data corresponding to the medical scan, for example,received from an expert user that later reviewed the contribution usagedata of the user and/or generated automatically by a subsystem. Theaccuracy score data 531 can include an aggregate accuracy scoregenerated automatically by a subsystem, for example, based on theaccuracy data of multiple contributions by the user over time.

The performance data can also include efficiency score data 532generated automatically by a subsystem for each contribution based on anamount of time taken to complete a contribution, for example, from atime the request for a contribution was sent to the client device to atime that the contribution was received from the client device, based ontiming data received from the client device itself, and/or based onother factors. The efficiency score can include an aggregate efficiencyscore, which can be generated automatically by a subsystem based on theindividual efficiency scores over time and/or based on determining acontribution completion rate, for example based on determining how manycontributions were completed in a fixed time window.

Aggregate performance score data 533 can be generated automatically by asubsystem based on the aggregate efficiency and/or accuracy data. Theaggregate performance data can include categorized performance data 534,for example, corresponding to different scan types, different anatomicalregions, different subsystems, different interactive interface featuresand/or display parameters. The categorized performance data 534 can bedetermined automatically by a subsystem based on the scan type data 421and/or anatomical region data 422 of the medical scan associated witheach contribution, one or more subsystems 101 associated with eachcontribution, and/or interactive interface feature data 524 associatedwith each contribution. The aggregate performance data can also be basedon performance score data 530 of individual employees if the usercorresponds to a medical entity, for example, retrieved based on userprofile identifiers 355 included in the employee data 513. Theperformance score data can also include ranking data 535, which caninclude an overall ranking or categorized rankings, for example,generated automatically by a subsystem or the database itself based onthe aggregate performance data.

In some embodiments, aggregate data for each user can be further brokendown based on scores for distinct scan categories, for example, based onthe scan classifier data 420, for example, where a first aggregate datascore is generated for a user “A” based on scores from all knee x-rays,and a second aggregate data score is generated for user A based onscores from all chest CT scans. Aggregate data for each user can befurther based on scores for distinct diagnosis categories, where a firstaggregate data score is generated for user A based on scores from allnormal scans, and a second aggregate data score is generated for user Abased on scores from all scans that contain an abnormality. This can befurther broken down, where a first aggregate score is generated for userA based on all scores from scans that contain an abnormality of a firsttype and/or in a first anatomical location, and a second aggregate scoreis generated for A based on all scores from scans that contain anabnormality of a second type and/or in a second location. Aggregate datafor each user can be further based on affiliation data, where a rankingis generated for a medical professional “B” based on scores from allmedical professionals with the same affiliation data, and/or where aranking is generated for a hospital “C” based on scores for allhospitals, all hospitals in the same geographical region, etc. Aggregatedata for each user can be further based on scores for interfacefeatures, where a first aggregate data score is generated for user Abased on scores using a first interface feature, and a second aggregatedata score is generated for user A based on scores using a firstinterface feature.

The user profile entry can include qualification data 540. Thequalification data can include experience data 541 such as educationdata, professional practice data, number of years practicing, awardsreceived, etc. The qualification data 540 can also include certificationdata 542 corresponding to certifications earned based on contributionsto one or more subsystems, for example, assigned to users automaticallyby a subsystem based on the performance score data 530 and/or based on anumber of contributions in the contribution usage data 523 and/ortraining usage data 522. For example, the certifications can correspondto standard and/or recognized certifications to train medicalprofessionals and/or incentivize medical professionals to use thesystem. The qualification data 540 can include expert data 543. Theexpert data 543 can include a binary expert identifier, which can begenerated automatically by a subsystem based on experience data 541,certification data 542, and/or the performance score data 530, and canindicate whether the user is an expert user. The expert data 543 caninclude a plurality of categorized binary expert identifierscorresponding to a plurality of qualification categories correspondingto corresponding to scan types, anatomical regions, and/or theparticular subsystems. The categorized binary expert identifiers can begenerated automatically by a subsystem based on the categorizedperformance data 534 and/or the experience data 541. The categories beranked by performance score in each category to indicate particularspecialties. The expert data 543 can also include an expert ranking orcategorized expert ranking with respect to all experts in the system.

The user profile entry can include subscription data 550, which caninclude a selected one of a plurality of subscription options that theuser has subscribed to. For example, the subscription options cancorrespond to allowed usage of one or more subsystems, such as a numberof times a user can utilize a subsystem in a month, and/or to acertification program, for example paid for by a user to receivetraining to earn a subsystem certification of certification data 542.The subscription data can include subscription expiration information,and/or billing information. The subscription data can also includesubscription status data 551, which can for example indicate a number ofremaining usages of a system and/or available credit information. Forexample, the remaining number of usages can decrease and/or availablecredit can decrease in response to usages that utilize one or moresubsystems as a service, for example, indicated in the consumption usagedata 521 and/or training usage data 522. In some embodiments, theremaining number of usages can increase and/or available credit canincrease in response to usages that correspond to contributions, forexample, based on the contribution usage data 523. An increase in creditcan be variable, and can be based on a determined quality of eachcontribution, for example, based on the performance score data 530corresponding to the contribution where a higher performance scorecorresponds to a higher increase in credit, based on scan priority data427 of the medical scan where contributing to higher priority scanscorresponds to a higher increase in credit, or based on other factors.

The user profile entry 354 can include interface preference data 560.The interface preference data can include a preferred interactiveinterface feature set 561, which can include one or more interactiveinterface feature identifiers and/or one or more interactive interfaceversion identifiers of interface feature entries 358 and/or versionidentifiers of the interface features. Some or all of the interfacefeatures of the preferred interactive interface feature set 561 cancorrespond to display parameter data 470 of medical scans. The preferredinteractive interface feature set 561 can include a single interactivefeature identifier for one or more feature types and/or interface types,and/or can include a single interactive interface version identifier forone or more interface categories. The preferred interactive interfacefeature set 561 can include a ranking of multiple features for the samefeature type and/or interface type. The ranked and/or unranked preferredinteractive interface feature set 561 can be generated based on userinput to an interactive interface of the client device to select and/orrank some or all of the interface features and/or versions. Some or allof the features and/or versions of the preferred interactive feature setcan be selected and/or ranked automatically by a subsystem such as themedical scan interface evaluator system, for example based on interfacefeature performance score data and/or feature popularity data.Alternatively or in addition, the performance score data 530 can beutilized by a subsystem to automatically determine the preferredinteractive feature set, for example, based on the scores in differentfeature-based categories of the categorized performance data 534.

The user profile entry 354 can include report formatting data 570, whichcan indicate report formatting preferences indicated by the user. Thiscan include font, text size, header data, bulleting or numbering type,margins, file type, preferences for including one or more full orcropped image slices 412, preferences for including similar medicalscans, preferences for including additional medical scans in reports, orother formatting preference to list natural language text data and/orimage data corresponding to each abnormality. Some or all of the reportformatting data 570 can be based on interface preference data 560. Thereport formatting data 570 can be used by one or more subsystems toautomatically generate report data 449 of medical scans based on thepreferences of the requesting user.

FIG. 5B presents an embodiment of a medical scan analysis function entry356, stored in medical scan analysis function database 346 or otherwiseassociated with one of a plurality of medical scan analysis functionstrained by and/or utilized by one or more subsystems 101. For example, amedical scan analysis function can include one or more medical scanimage analysis functions trained by the medical scan image analysissystem 112; one or more medical scan natural language analysis functionstrained by the medical scan natural language analysis system 114; one ormore medical scan similarity analysis function trained by the medicalscan image analysis system 112, the medical scan natural languageanalysis system 114, and/or the medical scan comparison system 116; oneor more medical report generator functions trained by the medical scannatural language analysis system 114 and/or the medical scan imageanalysis system 112, and/or the medical report analysis function trainedby the medical scan natural language analysis system 114. Some or all ofthe medical scan analysis functions can correspond to medical scaninference functions of the medical scan diagnosing system 108, thede-identification function and/or the inference functions utilized by amedical picture archive integration system as discussed in conjunctionwith FIGS. 8A-8F, or other functions and/or processes described hereinin conjunction with one or more subsystems 101. Each medical scananalysis function entry 356 can include a medical scan analysis functionidentifier 357.

A medical scan analysis function entry 356 can include functionclassifier data 610. Function classifier data 610 can include input andoutput types corresponding to the function. For example the functionclassifier data can include input scan category 611 that indicates whichtypes of scans can be used as input to the medical scan analysisfunction. For example, input scan category 611 can indicate that amedical scan analysis function is for chest CT scans from a particularhospital or other medical entity. The input scan category 611 caninclude one or more categories included in scan classifier data 420. Invarious embodiments, the input scan category 611 corresponds to thetypes of medical scans that were used to train the medical scan analysisfunction. Function classifier data 610 can also include output type data612 that characterizes the type of output that will be produced by thefunction, for example, indicating that a medical scan analysis functionis used to generate medical codes 447. The input scan category 611 canalso include information identifying which subsystems 101 areresponsible for running the medical scan analysis function.

A medical scan analysis function entry 356 can include trainingparameters 620. This can include training set data 621, which caninclude identifiers for the data used to train the medical scan analysisfunction, such as a set of medical scan identifiers 353 corresponding tothe medical scans used to train the medical scan analysis function, alist of medical scan reports and corresponding medical codes used totrain the medical scan analysis function, etc. Alternatively or inaddition to identifying particular scans of the training set, thetraining set data 621 can identify training set criteria, such asnecessary scan classifier data 420, necessary abnormality locations,classifiers, or other criteria corresponding to abnormality annotationdata 442, necessary confidence score data 460, for example, indicatingthat only medical scans with diagnosis data 440 assigned a truth flag461 or with confidence score data 460 otherwise comparing favorably to atraining set confidence score threshold are included, a number ofmedical scans to be included and proportion data corresponding todifferent criteria, or other criteria used to populate a training setwith data of medical scans. Training parameters 620 can include modeltype data 622 indicating one or more types of model, methods, and/ortraining functions used to determine the medical scan analysis functionby utilizing the training set 621. Training parameters 620 can includemodel parameter data 623 that can include a set of features of thetraining data selected to train the medical scan analysis function,determined values for weights corresponding to selected input and outputfeatures, determined values for model parameters corresponding to themodel itself, etc. The training parameter data can also include testingdata 624, which can identify a test set of medical scans or other dataused to test the medical scan analysis function. The test set can be asubset of training set 621, include completely separate data thantraining set 621, and/or overlap with training set 621. Alternatively orin addition, testing data 624 can include validation parameters such asa percentage of data that will be randomly or pseudo-randomly selectedfrom the training set for testing, parameters characterizing a crossvalidation process, or other information regarding testing. Trainingparameters 620 can also include training error data 625 that indicates atraining error associated with the medical scan analysis function, forexample, based on applying cross validation indicated in testing data624.

A medical scan analysis function entry 356 can include performance scoredata 630. Performance data can include model accuracy data 631, forexample, generated and/or updated based on the accuracy of the functionwhen performed on new data. For example, the model accuracy data 631 caninclude or be calculated based on the model error for determined forindividual uses, for example, generated by comparing the output of themedical scan analysis function to corresponding data generated by userinput to interactive interface 275 in conjunction with a subsystem 101and/or generated by comparing the output of the medical scan analysisfunction to medical scans with a truth flag 461. The model accuracy data631 can include aggregate model accuracy data computed based on modelerror of individual uses of the function over time. The performancescore data 630 can also include model efficiency data 632, which can begenerated based on how quickly the medical scan analysis functionperforms, how much memory is utilized by medical scan analysis function,or other efficiency data relating to the medical scan analysis function.Some or all of the performance score data 630 can be based on trainingerror data 625 or other accuracy and/or efficiency data determinedduring training and/or validation. As used herein, a “high” performancescore refers to a more favorable performance or rating than a “low”performance score.

A medical scan analysis function entry 356 can include version data 640.The version data can include a version identifier 641. The version datacan indicate one or more previous version identifiers 642, which can mapto version identifiers 641 stored in other medical scan analysisfunction entry 356 that correspond to previous versions of the function.Alternatively or in addition, the version data can indicate multipleversions of the same type based on function classifier data 610, canindicate the corresponding order and/or rank of the versions, and/or canindicate training parameters 620 associated with each version.

A medical scan analysis function entry 356 can include remediation data650. Remediation data 650 can include remediation instruction data 651which can indicate the steps in a remediation process indicating how amedical scan analysis function is taken out of commission and/orreverted to a previous version in the case that remediation isnecessary. The version data 640 can further include remediation criteriadata 652, which can include threshold data or other criteria used toautomatically determine when remediation is necessary. For example, theremediation criteria data 652 can indicate that remediation is necessaryat any time where the model accuracy data and/or the model efficiencydata compares unfavorably to an indicated model accuracy thresholdand/or indicated model efficiency threshold. The remediation data 650can also include recommissioning instruction data 653, identifyingrequired criteria for recommissioning a medical scan analysis functionand/or updating a medical scan analysis function. The remediation data650 can also include remediation history, indicating one or moreinstances that the medical scan analysis function was taken out ofcommission and/or was recommissioned.

FIGS. 6A and 6B present an embodiment of a medical scan diagnosingsystem 108.

The medical scan diagnosing system 108 can generate inference data 1110for medical scans by utilizing a set of medical scan inference functions1105, stored and run locally, stored and run by another subsystem 101,and/or stored in the medical scan analysis function database 346, wherethe function and/or parameters of the function can be retrieved from thedatabase by the medical scan diagnosing system. For example, the set ofmedical scan inference function 1105 can include some or all medicalscan analysis functions described herein or other functions thatgenerate inference data 1110 based on some or all data corresponding toa medical scan such as some or all data of a medical scan entry 352.Each medical scan inference function 1105 in the set can correspond to ascan category 1120, and can be trained on a set of medical scans thatcompare favorably to the scan category 1120. For example, each inferencefunction can be trained on a set of medical scans of the one or moresame scan classifier data 420, such as the same and/or similar scantypes, same and/or similar anatomical regions locations, same and/orsimilar machine models, same and/or similar machine calibration, sameand/or similar contrasting agent used, same and/or similar originatingentity, same and/or similar geographical region, and/or otherclassifiers. Thus, the scan categories 1120 can correspond to one ormore of a scan type, scan anatomical region data, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data 420. For example, a first medical scan inferencefunction can be directed to characterizing knee x-rays, and a secondmedical scan inference function can be directed to chest CT scans. Asanother example, a first medical scan inference function can be directedto characterizing CT scans from a first hospital, and a second medicalscan image analysis function can be directed to characterizing CT scansfrom a second hospital.

Training on these categorized sets separately can ensure each medicalscan inference function 1105 is calibrated according to its scancategory 1120, for example, allowing different inference functions to becalibrated on type specific, anatomical region specific, hospitalspecific, machine model specific, and/or region-specific tendenciesand/or discrepancies. Some or all of the medical scan inferencefunctions 1105 can be trained by the medical scan image analysis systemand/or the medical scan natural language processing system, and/or somemedical scan inference functions 1105 can utilize both image analysisand natural language analysis techniques to generate inference data1110. For example, some or all of the inference functions can utilizeimage analysis of the medical scan image data 410 and/or naturallanguage data extracted from abnormality annotation data 442 and/orreport data 449 as input, and generate diagnosis data 440 such asmedical codes 447 as output. Each medical scan inference function canutilize the same or different learning models to train on the same ordifferent features of the medical scan data, with the same or differentmodel parameters, for example indicated in the model type data 622 andmodel parameter data 623. Model type and/or parameters can be selectedfor a particular medical scan inference function based on particularcharacteristics of the one or more corresponding scan categories 1120,and some or all of the indicated in the model type data 622 and modelparameter data 623 can be selected automatically by a subsystem duringthe training process based on the particular learned and/or otherwisedetermined characteristics of the one or more corresponding scancategories 1120.

As shown in FIG. 6A, the medical scan diagnosing system 108 canautomatically select a medical scan for processing in response toreceiving it from a medical entity via the network. Alternatively, themedical scan diagnosing system 108 can automatically retrieve a medicalscan from the medical scan database that is selected based on a requestreceived from a user for a particular scan and/or based on a queue ofscans automatically ordered by the medical scan diagnosing system 108 oranother subsystem based on scan priority data 427.

Once a medical scan to be processed is determined, the medical scandiagnosing system 108 can automatically select an inference function1105 based on a determined scan category 1120 of the selected medicalscan and based on corresponding inference function scan categories. Thescan category 1120 of a scan can be determined based one some or all ofthe scan classifier data 420 and/or based on other metadata associatedwith the scan. This can include determining which one of the pluralityof medical scan inference functions 1105 matches or otherwise comparesfavorably to the scan category 1120, for example, by comparing the scancategory 1120 to the input scan category of the function classifier data610.

Alternatively or in addition, the medical scan diagnosing system 108 canautomatically determine which medical scan inference function 1105 isutilized based on an output preference that corresponding to a desiredtype of inference data 1110 that is outputted by an inference function1105. The output preference designated by a user of the medical scandiagnosing system 108 and/or based on the function of a subsystem 101utilizing the medical scan diagnosing system 108. For example, the setof inference functions 1105 can include inference functions that areutilized to indicate whether or not a medical scan is normal, toautomatically identify at least one abnormality in the scan, toautomatically characterize the at least one abnormality in the scan, toassign one or more medical codes to the scan, to generate naturallanguage text data and/or a formatted report for the scan, and/or toautomatically generate other diagnosis data such as some or all ofdiagnosis data 440 based on the medical scan. Alternatively or inaddition, some inference functions can also be utilized to automaticallygenerate confidence score data 460, display parameter data 470, and/orsimilar scan data 480. The medical scan diagnosing system 108 cancompare the output preference to the output type data 612 of the medicalscan inference function 1105 to determine the selected inferencefunction 1105. For example, this can be used to decide between a firstmedical scan inference function that automatically generates medicalcodes and a second medical scan inference function that automaticallygenerates natural language text for medical reports based on the desiredtype of inference data 1110.

Prior to performing the selected medical scan inference function 1105,the medical scan diagnosing system 108 can automatically perform aninput quality assurance function 1106 to ensure the scan classifier data420 or other metadata of the medical scan accurately classifies themedical scan such that the appropriate medical scan inference function1105 of the appropriate scan category 1120 is selected. The inputquality assurance function can be trained on, for example, medical scanimage data 410 of plurality of previous medical scans with verified scancategories. Thus, the input quality assurance function 1106 can takemedical scan image data 410 as input and can generate an inferred scancategory as output. The inferred scan category can be compared to thescan category 1120 of the scan, and the input quality assurance function1106 can determine whether or not the scan category 1120 is appropriateby determining whether the scan category 1120 compares favorably to theautomatically generated inferred scan category. The input qualityassurance function 1106 can also be utilized to reassign the generatedinferred scan category to the scan category 1120 when the scan category1120 compares favorably to the automatically generated inferred scancategory. The input quality assurance function 1106 can also be utilizedto assign the generated inferred scan category to the scan category 1120for incoming medical scans that do not include any classifying data,and/or to add classifiers in scan classifier data 420 to medical scansmissing one or more classifiers.

In various embodiments, upon utilizing the input quality assurancefunction 1106 to determine that the scan category 1120 determined by ascan classifier data 420 or other metadata is inaccurate, the medicalscan diagnosing system 108 can transmit an alert and/or an automaticallygenerated inferred scan category to the medical entity indicating thatthe scan is incorrectly classified in the scan classifier data 420 orother metadata. In some embodiments, the medical scan diagnosing system108 can automatically update performance score data corresponding to theoriginating entity of the scan indicated in originating entity data 423,or another user or entity responsible for classifying the scan, forexample, where a lower performance score is generated in response todetermining that the scan was incorrectly classified and/or where ahigher performance score is generated in response to determining thatthe scan was correctly classified.

In some embodiments, the medical scan diagnosing system 108 can transmitthe medical scan and/or the automatically generated inferred scancategory to a selected user. The user can be presented the medical scanimage data 410 and/or other data of the medical scan via the interactiveinterface 275, for example, displayed in conjunction with the medicalscan assisted review system 102. The interface can prompt the user toindicate the appropriate scan category 1120 and/or prompt the user toconfirm and/or edit the inferred scan category, also presented to theuser. For example, scan review data can be automatically generated toreflect the user generated and/or verified scan category 1120, This userindicated scan category 1120 can be utilized to select to the medicalscan inference function 1105 and/or to update the scan classifier data420 or other metadata accordingly. In some embodiments, for example,where the scan review data indicates that the selected user disagreeswith the automatically generated inferred scan category created by theinput quality assurance function 1106, the medical scan diagnosingsystem 108 can automatically update performance score data 630 of theinput quality assurance function 1106 by generating a low performancescore and/or determine to enter the remediation step 1140 for the inputquality assurance function 1106.

The medical scan diagnosing system 108 can also automatically perform anoutput quality assurance step after a medical scan inference function1105 has been performed on a medical scan to produce the inference data1110, as illustrated in the embodiment presented in FIG. 6B. The outputquality assurance step can be utilized to ensure that the selectedmedical scan inference function 1105 generated appropriate inferencedata 1110 based on expert feedback. The inference data 1110 generated byperforming the selected medical scan inference function 1105 can be sentto a client device 120 of a selected expert user, such as an expert userin the user database selected based on categorized performance dataand/or qualification data that corresponds to the scan category 1120and/or the inference itself, for example, by selecting an expert userbest suited to review an identified abnormality classifier category 444and/or abnormality pattern category 446 in the inference data 1110 basedon categorized performance data and/or qualification data of acorresponding user entry. The selected user can also correspond to amedical professional or other user employed at the originating entityand/or corresponding to the originating medical professional, indicatedin the originating entity data 423.

FIG. 6B illustrates an embodiment of the medical scan diagnosing system108 in conjunction with performing a remediation step 1140. The medicalscan diagnosing system 108 can monitor the performance of the set ofmedical scan inference functions 1105, for example, based on evaluatinginference accuracy data outputted by an inference data evaluationfunction and/or based monitoring on the performance score data 630 inthe medical scan analysis function database, and can determine whetheror not if the corresponding medical scan inference function 1105 isperforming properly. This can include, for example, determining if aremediation step 1140 is necessary for a medical scan inference function1105, for example, by comparing the performance score data 630 and/orinference accuracy data to remediation criteria data 652. Determining ifa remediation step 1140 is necessary can also be based on receiving anindication from the expert user or another user that remediation isnecessary for one or more identified medical scan inference functions1105 and/or for all of the medical scan inference functions 1105.

In various embodiments, a remediation evaluation function is utilized todetermine if a remediation step 1140 is necessary for medical scaninference function 1105. The remediation evaluation function can includedetermining that remediation is necessary when recent accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below the normal performance level of the particular inferencefunction. The remediation evaluation function can include determiningthat remediation is necessary when recent or overall accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below a recent or overall average for all or similar medicalscan inference functions 1105. The remediation evaluation function caninclude determining that remediation is necessary only after a thresholdnumber of incorrect diagnoses are made. In various embodiments, multiplethreshold number of incorrect diagnoses correspond to differentdiagnoses categories. For example, the threshold number of incorrectdiagnoses for remediation can be higher for false negative diagnosesthan false positive diagnoses. Similarly, categories corresponding todifferent diagnosis severities and/or rarities can have differentthresholds, for example where a threshold number of more severe and/ormore rare diagnoses that were inaccurate to necessitate remediation islower than a threshold number of less severe and/or less rare diagnosesthat were inaccurate.

The remediation step 1140 can include automatically updating anidentified medical inference function 1105. This can includeautomatically retraining identified medical inference function 1105 onthe same training set or on a new training set that includes new data,data with higher corresponding confidence scores, or data selected basedon new training set criteria. The identified medical inference function1105 can also be updated and/or changed based on the review datareceived from the client device. For example, the medical scan andexpert feedback data can be added to the training set of the medicalscan inference function 1105, and the medical scan inference function1105 can be retrained on the updated training set. Alternatively or inaddition, the expert user can identify additional parameters and/orrules in the expert feedback data based on the errors made by theinference function in generating the inference data 1110 for the medicalscan, and these parameters and/or rules can be applied to update themedical scan inference function, for example, by updating the model typedata 622 and/or model parameter data 623.

The remediation step 1140 can also include determining to split a scancategory 1120 into two or more subcategories. Thus, two or more newmedical scan inference functions 1105 can be created, where each newmedical scan inference functions 1105 is trained on a correspondingtraining set that is a subset of the original training set and/orincludes new medical scan data corresponding to the subcategory. Thiscan allow medical scan inference functions 1105 to become morespecialized and/or allow functions to utilize characteristics and/ordiscrepancies specific to the subcategory when generating inference data1110. Similarly, a new scan category 1120 that was not previouslyrepresented by any of the medical scan inference functions 1105 can beadded in the remediation step, and a new medical scan inferencefunctions 1105 can be trained on a new set of medical scan data thatcorresponds to the new scan category 1120. Splitting a scan categoryand/or adding a scan category can be determined automatically by themedical scan diagnosing system 108 when performing the remediation step1140, for example, based on performance score data 630. This can also bedetermined based on receiving instructions to split a category and/oradd a new scan category from the expert user or other user of thesystem.

After a medical scan inference function 1105 is updated or created forthe first time, the remediation step 1140 can further undergo acommissioning test, which can include rigorous testing of the medicalscan inference function 1105 on a testing set, for example, based on thetraining parameters 620. For example, the commissioning test can bepassed when the medical scan inference function 1105 generates athreshold number of correct inference data 1110 and/or the test can bepassed if an overall or average discrepancy level between the inferencedata and the test data is below a set error threshold. The commissioningtest can also evaluate efficiency, where the medical scan inferencefunction 1105 only passes the commissioning test if it performs at orexceeds a threshold efficiency level. If the medical scan inferencefunction 1105 fails the commissioning test, the model type and/or modelparameters can be modified automatically or based on user input, and themedical scan inference function can be retested, continuing this processuntil the medical scan inference function 1105 passes the commissioningtest.

The remediation step 1140 can include decommissioning the medical scaninference function 1105, for example, while the medical scan inferencefunction is being retrained and/or is undergoing the commissioning test.Incoming scans to the medical scan diagnosing system 108 with a scancategory 1120 corresponding to a decommissioned medical scan inferencefunction 1105 can be sent directly to review by one or more users, forexample, in conjunction with the medical scan annotator system 106.These user-reviewed medical scans and corresponding annotations can beincluded in an updated training set used to train the decommissionedmedical scan inference function 1105 as part of the remediation step1140. In some embodiments, previous versions of the plurality of medicalscan image analysis functions can be stored in memory of the medicalscan diagnosing system and/or can be determined based on the versiondata 640 of a medical scan inference function 1105. A previous versionof a medical scan inference function 1105, such as most recent versionor version with the highest performance score, can be utilized duringthe remediation step 1140 as an alternative to sending all medical scansto user review.

A medical scan inference function can also undergo the remediation step1140 automatically in response to a hardware and/or software update onprocessing, memory, and/or other computing devices where the medicalscan inference function 1105 is stored and/or performed. Differentmedical scan inference functions 1105 can be containerized on their owndevices by utilizing a micro-service architecture, so hardware and/orsoftware updates may only necessitate that one of the medical scaninference functions 1105 undergo the remediation step 1140 while theothers remain unaffected. A medical scan inference function 1105 canalso undergo the remediation step 1140 automatically in response tonormal system boot-up, and/or periodically in fixed intervals. Forexample, in response to a scheduled or automatically detected hardwareand/or software update, change, or issue, one or more medical scaninference functions 1105 affected by this hardware or software can betaken out of commission until they each pass the commissioning test.Such criteria can be indicated in the remediation criteria data 652.

The medical scan diagnosing system 108 can automatically manage usagedata, subscription data, and/or billing data for the plurality of userscorresponding to user usage of the system, for example, by utilizing,generating, and/or updating some or all of the subscription data of theuser database. Users can pay for subscriptions to the system, which caninclude different subscription levels that can correspond to differentcosts. For example, a hospital can pay a monthly cost to automaticallydiagnose up to 100 medical scans per month. The hospital can choose toupgrade their subscription or pay per-scan costs for automaticdiagnosing of additional scans received after the quota is reachedand/or the medical scan diagnosing system 108 can automatically sendmedical scans received after the quota is reached to an expert userassociated with the hospital. In various embodiments incentive programscan be used by the medical scan diagnosing system to encourage expertsto review medical scans from different medical entities. For example, anexpert can receive credit to their account and/or subscription upgradesfor every medical scan reviewed, or after a threshold number of medicalscans are reviewed. The incentive programs can include interactions by auser with other subsystems, for example, based on contributions made tomedical scan entries via interaction with other subsystems.

FIG. 7A presents an embodiment of a medical scan image analysis system112. A training set of medical scans used to train one more medical scanimage analysis functions can be received from one or more client devicesvia the network and/or can be retrieved from the medical scan database342, for example, based on training set data 621 corresponding tomedical scan image analysis functions. Training set criteria, forexample, identified in training parameters 620 of the medical scan imageanalysis function, can be utilized to automatically identify and selectmedical scans to be included in the training set from a plurality ofavailable medical scans. The training set criteria can be automaticallygenerated based on, for example, previously learned criteria, and/ortraining set criteria can be received via the network, for example, froman administrator of the medical scan image analysis system. The trainingset criteria can include a minimum training set size. The training setcriteria can include data integrity requirements for medical scans inthe training set such as requiring that the medical scan is assigned atruth flag 461, requiring that performance score data for a hospitaland/or medical professional associated with the medical scan comparesfavorably to a performance score threshold, requiring that the medicalscan has been reviewed by at least a threshold number of medicalprofessionals, requiring that the medical scan and/or a diagnosiscorresponding to a patient file of the medical scan is older than athreshold elapsed time period, or based on other criteria intended toinsure that the medical scans and associated data in the training set isreliable enough to be considered “truth” data. The training set criteriacan include longitudinal requirements such the number of requiredsubsequent medical scans for the patient, multiple required types ofadditional scans for the patient, and/or other patient filerequirements.

The training set criteria can include quota and/or proportionrequirements for one or more medical scan classification data. Forexample, the training set criteria can include meeting quota and/orproportion requirements for one or more scan types and/or human bodylocation of scans, meeting quota or proportion requirements for a numberof normal medical scans and a number of medicals scans with identifiedabnormalities, meeting quota and/or proportion requirements for a numberof medical scans with abnormalities in certain locations and/or a numberof medical scans with abnormalities that meet certain size, type, orother characteristics, meeting quota and/or proportion data for a numberof medical scans with certain diagnosis or certain corresponding medicalcodes, and/or meeting other identified quota and/or proportion datarelating to metadata, patient data, or other data associated with themedical scans.

In some embodiments, multiple training sets are created to generatecorresponding medical scan image analysis functions, for example,corresponding to some or all of the set of medical scan inferencefunctions 1105. Some or all training sets can be categorized based onsome or all of the scan classifier data 420 as described in conjunctionwith the medical scan diagnosing system 108, where medical scans areincluded in a training set based on their scan classifier data 420matching the scan category of the training set. In some embodiments, theinput quality assurance function 1106 or another input check step can beperformed on medical scans selected for each training set to confirmthat their corresponding scan classifier data 420 is correct. In someembodiments, the input quality assurance function can correspond to itsown medical scan image analysis function, trained by the medical scanimage analysis system, where the input quality assurance functionutilizes high level computer vision technology to determine a scancategory 1120 and/or to confirm the scan classifier data 420 alreadyassigned to the medical scan.

In some embodiments, the training set will be used to create a singleneural network model, or other model corresponding to model type data622 and/or model parameter data 623 of the medical scan image analysisfunction that can be trained on some or all of the medical scanclassification data described above and/or other metadata, patient data,or other data associated with the medical scans. In other embodiments, aplurality of training sets will be created to generate a plurality ofcorresponding neural network models, where the multiple training setsare divided based on some or all of the medical scan classification datadescribed above and/or other metadata, patient data, or other dataassociated with the medical scans. Each of the plurality of neuralnetwork models can be generated based on the same or different learningalgorithm that utilizes the same or different features of the medicalscans in the corresponding one of the plurality of training sets. Themedical scan classifications selected to segregate the medical scansinto multiple training sets can be received via the network, for examplebased on input to an administrator client device from an administrator.The medical scan classifications selected to segregate the medical scanscan be automatically determined by the medical scan image analysissystem, for example, where an unsupervised clustering algorithm isapplied to the original training set to determine appropriate medicalscan classifications based on the output of the unsupervised clusteringalgorithm.

In embodiments where the medical scan image analysis system is used inconjunction with the medical scan diagnosing system, each of the medicalscan image analysis functions associated with each neural network modelcan correspond to one of the plurality of neural network modelsgenerated by the medical scan image analysis system. For example, eachof the plurality of neural network models can be trained on a trainingset classified on scan type, scan human body location, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data as discussed in conjunction with the medical scandiagnosing system. In embodiments where the training set classifiers arelearned, the medical scan diagnosing system can determine which of themedical scan image analysis functions should be applied based on thelearned classifying criteria used to segregate the original trainingset.

A computer vision-based learning algorithm used to create each neuralnetwork model can include selecting a three-dimensional subregion 1310for each medical scan in the training set. This three-dimensionalsubregion 1310 can correspond to a region that is “sampled” from theentire scan that may represent a small fraction of the entire scan.Recall that a medical scan can include a plurality of orderedcross-sectional image slices. Selecting a three-dimensional subregion1310 can be accomplished by selecting a proper image slice subset 1320of the plurality of cross-sectional image slices from each of theplurality of medical scans, and by further selecting a two-dimensionalsubregion 1330 from each of the selected subset of cross-sectional imageslices of the each of the medical scans. In some embodiments, theselected image slices can include one or more non-consecutive imageslices and thus a plurality of disconnected three-dimensional subregionswill be created. In other embodiments, the selected proper subset of theplurality of image slices correspond to a set of consecutive imageslices, as to ensure that a single, connected three-dimensionalsubregion is selected. In some embodiments, entire scans of the trainingset are used to train the neural network model. In such embodiment, asused herein, the three-dimensional subregion 1310 can refer to all ofthe medical scan image data 410 of a medical scan.

In some embodiments, a density windowing step can be applied to the fullscan or the selected three-dimensional subregion. The density windowingstep can include utilizing a selected upper density value cut off and/ora selected lower density value cut off, and masking pixels with highervalues than the upper density value cut off and/or masking pixels withlower values than the lower density value cut off. The upper densityvalue cut off and/or a selected lower density value cut off can bedetermined based on based on the range and/or distribution of densityvalues included in the region that includes the abnormality, and/orbased on the range and/or distribution of density values associated withthe abnormality itself, based on user input to a subsystem, based ondisplay parameter data associated with the medical scan or associatedwith medical scans of the same type, and/or can be learned in thetraining step. In some embodiments, a non-linear density windowingfunction can be applied to alter the pixel density values, for example,to stretch or compress contrast. In some embodiments, this densitywindowing step can be performed as a data augmenting step, to createadditional training data for a medical scan in accordance with differentdensity windows.

Having determined the subregion training set 1315 of three-dimensionalsubregions 1310 corresponding to the set of full medical scans in thetraining set, the medical scan image analysis system can complete atraining step 1352 by performing a learning algorithm on the pluralityof three-dimensional subregions to generate model parameter data 1355 ofa corresponding learning model. The learning model can include one ormore of a neural network, an artificial neural network, a convolutionalneural network, a Bayesian model, a support vector machine model, acluster analysis model, or other supervised or unsupervised learningmodel. The model parameter data 1355 can generated by performing thelearning algorithm 1350, and the model parameter data 1355 can beutilized to determine the corresponding medical scan image analysisfunctions. For example, some or all of the model parameter data 1355 canbe mapped to the medical scan analysis function in the model parameterdata 623 or can otherwise define the medical scan analysis function.

The training step 1352 can include creating feature vectors for eachthree-dimensional subregion of the training set for use by the learningalgorithm 1350 to generate the model parameter data 1355. The featurevectors can include the pixel data of the three-dimensional subregionssuch as density values and/or grayscale values of each pixel based on adetermined density window. The feature vectors can also include otherfeatures as additional input features or desired output features, suchas known abnormality data such as location and/or classification data,patient history data such as risk factor data or previous medical scans,diagnosis data, responsible medical entity data, scan machinery model orcalibration data, contrast agent data, medical code data, annotationdata that can include raw or processed natural language text data, scantype and/or anatomical region data, or other data associated with theimage, such as some or all data of a medical scan entry 352. Featurescan be selected based on administrator instructions received via thenetwork and/or can be determined based on determining a feature set thatreduces error in classifying error, for example, by performing across-validation step on multiple models created using different featuresets. The feature vector can be split into an input feature vector andoutput feature vector. The input feature vector can include data thatwill be available in subsequent medical scan input, which can includefor example, the three-dimensional subregion pixel data and/or patienthistory data. The output feature vector can include data that will beinferred in in subsequent medical scan input and can include singleoutput value, such as a binary value indicating whether or not themedical scan includes an abnormality or a value corresponding to one ofa plurality of medical codes corresponding to the image. The outputfeature vector can also include multiple values which can includeabnormality location and/or classification data, diagnosis data, orother output. The output feature vector can also include a determinedupper density value cut off and/or lower density value cut off, forexample, characterizing which pixel values were relevant to detectingand/or classifying an abnormality. Features included in the outputfeature vector can be selected to include features that are known in thetraining set, but may not be known in subsequent medical scans such astriaged scans to be diagnosed by the medical scan diagnosing system,and/or scans to be labeled by the medical scan report labeling system.The set of features in the input feature vector and output featurevector, as well as the importance of different features where eachfeature is assigned a corresponding weight, can also be designated inthe model parameter data 1355.

Consider a medical scan image analysis function that utilizes a neuralnetwork. The neural network can include a plurality of layers, whereeach layer includes a plurality of neural nodes. Each node in one layercan have a connection to some or all nodes in the next layer, where eachconnection is defined by a weight value. Thus, the model parameter data1355 can include a weight vector that includes weight values for everyconnection in the network. Alternatively or in addition, the modelparameter data 1355 can include any vector or set of parametersassociated with the neural network model, which can include an upperdensity value cut off and/or lower density value cut off used to masksome of the pixel data of an incoming image, kernel values, filterparameters, bias parameters, and/or parameters characterizing one ormore of a plurality of convolution functions of the neural networkmodel. The medical scan image analysis function can be utilized toproduce the output vector as a function of the input feature vector andthe model parameter data 1355 that characterizes the neural networkmodel. In particular, the medical scan image analysis function caninclude performing a forward propagation step plurality of neuralnetwork layers to produce an inferred output vector based on the weightvector or other model parameter data 1355. Thus, the learning algorithm1350 utilized in conjunction with a neural network model can includedetermining the model parameter data 1355 corresponding to the neuralnetwork model, for example, by populating the weight vector with optimalweights that best reduce output error.

In particular, determining the model parameter data 1355 can includeutilizing a backpropagation strategy. The forward propagation algorithmcan be performed on at least one input feature vector corresponding toat least one medical scan in the training set to propagate the at leastone input feature vector through the plurality of neural network layersbased on initial and/or default model parameter data 1355, such as aninitial weight vector of initial weight values set by an administratoror chosen at random. The at least one output vector generated byperforming the forward propagation algorithm on the at least one inputfeature vector can be compared to the corresponding at least one knownoutput feature vector to determine an output error. Determining theoutput error can include, for example, computing a vector distance suchas the Euclidian distance, or squared Euclidian distance, between theproduced output vector and the known output vector, and/or determiningan average output error such as an average Euclidian distance or squaredEuclidian distance if multiple input feature vectors were employed.Next, gradient descent can be performed to determine an updated weightvector based on the output error or average output error. This gradientdescent step can include computing partial derivatives for the errorwith respect to each weight, or other parameter in the model parameterdata 1355, at each layer starting with the output layer. Chain rule canbe utilized to iteratively compute the gradient with respect to eachweight or parameter at each previous layer until all weight's gradientsare computed. Next updated weights, or other parameters in the modelparameter data 1355, are generated by updating each weight based on itscorresponding calculated gradient. This process can be repeated on atleast one input feature vector, which can include the same or differentat least one feature vector used in the previous iteration, based on theupdated weight vector and/or other updated parameters in the modelparameter data 1355 to create a new updated weight vector and/or othernew updated parameters in the model parameter data 1355. This processcan continue to repeat until the output error converges, the outputerror is within a certain error threshold, or another criterion isreached to determine the most recently updated weight vector and/orother model parameter data 1355 is optimal or otherwise determined forselection.

Having determined the medical scan neural network and its final othermodel parameter data 1355, an inference step 1354 can be performed onnew medical scans to produce inference data 1370, such as inferredoutput vectors, as shown in FIG. 7B. The inference step can includeperforming the forward propagation algorithm to propagate an inputfeature vector through a plurality of neural network layers based on thefinal model parameter data 1355, such as the weight values of the finalweight vector, to produce the inference data. This inference step 1354can correspond to performing the medical scan image analysis function,as defined by the final model parameter data 1355, on new medical scansto generate the inference data 1370, for example, in conjunction withthe medical scan diagnosing system 108 to generate inferred diagnosisdata or other selected output data for triaged medical scans based onits corresponding the input feature vector.

The inference step 1354 can include applying the density windowing stepto new medical scans. Density window cut off values and/or a non-lineardensity windowing function that are learned can be automatically appliedwhen performing the inference step. For example, if the training step1352 was used to determine optimal upper density value cut off and/orlower density value cut off values to designate an optimal densitywindow, the inference step 1354 can include masking pixels of incomingscans that fall outside of this determined density window beforeapplying the forward propagation algorithm. As another example, iflearned parameters of one or more convolutional functions correspond tothe optimal upper density value cut off and/or lower density value cutoff values, the density windowing step is inherently applied when theforward propagation algorithm is performed on the new medical scans.

In some embodiments where a medical scan analysis function is defined bymodel parameter data 1355 corresponding to a neutral network model, theneural network model can be a fully convolutional neural network. Insuch embodiments, only convolution functions are performed to propagatethe input feature vector through the layers of the neural network in theforward propagation algorithm. This enables the medical scan imageanalysis functions to process input feature vectors of any size. Forexample, as discussed herein, the pixel data corresponding to thethree-dimensional subregions is utilized input to the forwardpropagation algorithm when the training step 1352 is employed topopulate the weight vector and/or other model parameter data 1355.However, when performing the forward propagation algorithm in theinference step 1354, the pixel data of full medical scans can beutilized as input, allowing the entire scan to be processed to detectand/or classify abnormalities, or otherwise generate the inference data1370. This may be a preferred embodiment over other embodiments wherenew scans must also be sampled by selecting a three-dimensionalsubregions and/or other embodiments where the inference step requires“piecing together” inference data 1370 corresponding to multiplethree-dimensional subregions processed separately.

The inferred output vector of the inference data 1370 can include aplurality of abnormality probabilities mapped to a pixel location ofeach of a plurality of cross-sectional image slices of the new medicalscan. For example, the inferred output vector can indicate a set ofprobability matrices 1371, where each matrix in the set corresponds toone of the plurality of image slices of the medical scan, where eachmatrix is a size corresponding to the number of pixels in each imageslice, where each cell of each matrix corresponds to a pixel of thecorresponding image slice, whose value is the abnormality probability ofthe corresponding pixel.

A detection step 1372 can include determining if an abnormality ispresent in the medical scan based on the plurality of abnormalityprobabilities. Determining if an abnormality is present can include, forexample, determining that a cluster of pixels in the same region of themedical scan correspond to high abnormality probabilities, for example,where a threshold proportion of abnormality probabilities must meet orexceed a threshold abnormality probability, where an average abnormalityprobability of pixels in the region must meet or exceed a thresholdabnormality probability, where the region that includes the cluster ofpixels must be at least a certain size, etc. Determining if anabnormality is present can also include calculating a confidence scorebased on the abnormality probabilities and/or other data correspondingto the medical scan such as patient history data. The location of thedetected abnormality can be determined in the detection step 1372 basedon the location of the pixels with the high abnormality probabilities.The detection step can further include determining an abnormality region1373, such as a two-dimensional subregion on one or more image slicesthat includes some or all of the abnormality. The abnormality region1373 determined in the detection step 1372 can be mapped to the medicalscan to populate some or all of the abnormality location data 443 foruse by one or more other subsystems 101 and/or client devices 120.Furthermore, determining whether or not an abnormality exists in thedetection step 1372 can be used to populate some or all of the diagnosisdata 440 of the medical scan, for example, to indicate that the scan isnormal or contains an abnormality in the diagnosis data 440.

An abnormality classification step 1374 can be performed on a medicalscan in response to determining an abnormality is present.Classification data 1375 corresponding to one or more classificationcategories such as abnormality size, volume, pre-post contract, doublingtime, calcification, components, smoothness, texture, diagnosis data,one or more medical codes, a malignancy rating such as a Lung-RADSscore, or other classifying data as described herein can be determinedbased on the detected abnormality. The classification data 1375generated by the abnormality classification step 1374 can be mapped tothe medical scan to populate some or all of the abnormalityclassification data 445 of the corresponding abnormality classifiercategories 444 and/or abnormality pattern categories 446 and/or todetermine one or more medical codes 447 of the medical scan. Theabnormality classification step 1374 can include performing anabnormality classification function on the full medical scan, or theabnormality region 1373 determined in the detection step 1372. Theabnormality classification function can be based on another modeltrained on abnormality data such as a support vector machine model,another neural network model, or any supervised classification modeltrained on medical scans, or portions of medical scans, that includeknown abnormality classifying data to generate inference data for someor all of the classification categories. For example, the abnormalityclassification function can include another medical scan analysisfunction. Classification data 1375 in each of a plurality ofclassification categories can also be assigned their own calculatedconfidence score, which can also be generated by utilizing theabnormality classification function. Output to the abnormalityclassification function can also include at least one identified similarmedical scan and/or at least one identified similar cropped image, forexample, based on the training data. The abnormality classification stepcan also be included in the inference step 1354, where the inferredoutput vector or other inference data 1370 of the medical scan imageanalysis function includes the classification data 1375.

The abnormality classification function can be trained on full medicalscans and/or one or more cropped or full selected image slices frommedical scans that contain an abnormality. For example, the abnormalityclassification function can be trained on a set of two-dimensionalcropped slices that include abnormalities. The selected image slicesand/or the cropped region in each selected image slice for each scan inthe training set can be automatically selected based upon the knownlocation of the abnormality. Input to the abnormality classificationfunction can include the full medical scan, one or more selected fullimage slices, and/or one or more selected image slices cropped based ona selected region. Thus, the abnormality classification step can includeautomatically selecting one or more image slices that include thedetected abnormality. The slice selection can include selecting thecenter slice in a set of consecutive slices that are determined toinclude the abnormality or selecting a slice that has the largestcross-section of the abnormality, or selecting one or more slices basedon other criteria. The abnormality classification step can also includeautomatically generating one or more cropped two-dimensional imagescorresponding to the one or more of the selected image slices based onan automatically selected region that includes the abnormality.

Input to the abnormality classification function can also include otherdata associated with the medical scan, including patient history, riskfactors, or other metadata. The abnormality classification step can alsoinclude determining some or all of the characteristics based on data ofthe medical scan itself. For example, the abnormality size and volumecan be determined based on a number of pixels determined to be part ofthe detected abnormality. Other classifiers such as abnormality textureand/or smoothness can be determined by performing one or more otherpreprocessing functions on the image specifically designed tocharacterize such features. Such preprocessed characteristics can beincluded in the input to the abnormality classification function to themore difficult task of assigning a medical code or generating otherdiagnosis data. The training data can also be preprocessed to includesuch preprocessed features.

A similar scan identification step 1376 can also be performed on amedical scan with a detected abnormality and/or can be performed on theabnormality region 1373 determined in the detection step 1372. Thesimilar scan identification step 1376 can include generating similarabnormality data 1377, for example, by identifying one or more similarmedical scans or one or more similar cropped two-dimensional images froma database of medical scans and/or database of cropped two-dimensionalimages. Similar medical scans and/or cropped images can include medicalscans or cropped images that are visually similar, medical scans orcropped images that have known abnormalities in a similar location to aninferred abnormality location of the given medical scan, medical scansthat have known abnormalities with similar characteristics to inferredcharacteristics of an abnormality in the given scan, medical scans withsimilar patient history and/or similar risk factors, or some combinationof these factors and/or other known and/or inferred factors. The similarabnormality data 1377 can be mapped to the medical scan to populate someor all of its corresponding similar scan data 480 for use by one or moreother subsystems 101 and/or client devices 120.

The similar scans identification step 1376 can include performing a scansimilarity algorithm, which can include generating a feature vector forthe given medical scan and for medical scans in the set of medicalscans, where the feature vector can be generated based on quantitativeand/or category based visual features, inferred features, abnormalitylocation and/or characteristics such as the predetermined size and/orvolume, patient history and/or risk factor features, or other known orinferred features. A medical scan similarity analysis function can beapplied to the feature vector of the given medical scan and one or morefeature vectors of medical scans in the set. The medical scan similarityanalysis function can include computing a similarity distance such asthe Euclidian distance between the feature vectors, and assigning thesimilarity distance to the corresponding medical scan in the set.Similar medical scans can be identified based on determining one or moremedical scans in the set with a smallest computed similarity distance,based on ranking medical scans in the set based on the computedsimilarity distances and identifying a designated number of top rankedmedical scans, and/or based on determining if a similarity distancebetween the given medical scan and a medical scan in the set is smallerthan a similarity threshold. Similar medical scans can also beidentified based on determining medical scans in a database that mappedto a medical code that matches the medical code of the medical scan, ormapped to other matching classifying data. A set of identified similarmedical scans can also be filtered based on other inputted orautomatically generated criteria, where for example only medical scanswith reliable diagnosis data or rich patient reports, medical scans withcorresponding with longitudinal data in the patient file such asmultiple subsequent scans taken at later dates, medical scans withpatient data that corresponds to risk factors of the given patient, orother identified criteria, where only a subset of scans that comparefavorably to the criteria are selected from the set and/or only ahighest ranked single scan or subset of scans are selected from the set,where the ranking is automatically computed based on the criteria.Filtering the similar scans in this fashion can include calculating, orcan be based on previously calculated, one or more scores as discussedherein. For example, the ranking can be based on a longitudinal qualityscore, such as the longitudinal quality score 434, which can becalculated for an identified medical scan based on a number ofsubsequent and/or previous scans for the patient. Alternatively or inaddition, the ranking can be based on a confidence score associated withdiagnosis data of the scan, such as confidence score data 460, based onperformance score data associated with a user or medical entityassociated with the scan, based on an amount of patient history data ordata in the medical scan entry 352, or other quality factors. Theidentified similar medical scans can be filtered based on ranking thescans based on their quality score and/or based on comparing theirquality score to a quality score threshold. In some embodiments, alongitudinal threshold must be reached, and only scans that comparefavorably to the longitudinal threshold will be selected. For example,only scans with at least three scans on file for the patient and finalbiopsy data will be included.

In some embodiments, the similarity algorithm can be utilized inaddition to or instead of the trained abnormality classificationfunction to determine some or all of the inferred classification data1375 of the medical scan, based on the classification data such asabnormality classification data 445 or other diagnosis data 440 mappedto one or more of the identified similar scans. In other embodiments,the similarity algorithm is merely used to identify similar scans forreview by medical professionals to aid in review, diagnosis, and/orgenerating medical reports for the medical image.

A display parameter step 1378 can be performed based on the detectionand/or classification of the abnormality. The display parameter step caninclude generating display parameter data 1379, which can includeparameters that can be used by an interactive interface to best displayeach abnormality. The same or different display parameters can begenerated for each abnormality. The display parameter data generated inthe display parameter step 1378 can be mapped to the medical scan topopulate some or all of its corresponding display parameter data 470 foruse by one or more other subsystems 101 and/or client devices 120.

Performing the display parameter step 1378 can include selecting one ormore image slices that include the abnormality by determining the one ormore image slices that include the abnormality and/or determining one ormore image slices that has a most optimal two-dimensional view of theabnormality, for example by selecting the center slice in a set ofconsecutive slices that are determined to include the abnormality,selecting a slice that has the largest cross-section of the abnormality,selecting a slice that includes a two-dimensional image of theabnormality that is most similar to a selected most similartwo-dimensional-image, selecting the slice that was used as input to theabnormality classification step and/or similar scan identification step,or based on other criteria. This can also include automatically croppingone or more selected image slices based on an identified region thatincludes the abnormality. This can also select an ideal Hounsfieldwindow that best displays the abnormality. This can also includeselecting other display parameters based on data generated by themedical scan interface evaluating system and based on the medical scan.

FIGS. 8A-8F illustrate embodiments of a medical picture archiveintegration system 2600. The medical picture archive integration system2600 can provide integration support for a medical picture archivesystem 2620, such as a PACS that stores medical scans. The medicalpicture archive integration system 2600 can utilize model parametersreceived from a central server system 2640 via a network 2630 to performan inference function on de-identified medical scans of medical scansreceived from the medical picture archive system 2620. The annotationdata produced by performing the inference function can be transmittedback to the medical picture archive system. Furthermore, the annotationdata and/or de-identified medical scans can be sent to the centralserver system 2640, and the central server system can train on thisinformation to produce new and/or updated model parameters fortransmission back to the medical picture archive integration system 2600for use on subsequently received medical scans.

In various embodiments, medical picture archive integration system 2600includes a de-identification system that includes a first memorydesignated for protected health information (PHI), operable to perform ade-identification function on a DICOM image, received from a medicalpicture archive system, to identify at least one patient identifier andgenerate a de-identified medical scan that does not include the at leastone patient identifier. The medical picture archive integration systemfurther includes a de-identified image storage system that stores thede-identified medical scan in a second memory that is separate from thefirst memory, and an annotating system, operable to utilize modelparameters received from a central server to perform an inferencefunction on the de-identified medical scan, retrieved from the secondmemory to generate annotation data for transmission to the medicalpicture archive system as an annotated DICOM file.

The first memory and the second memory can be implemented by utilizingseparate storage systems: the first memory can be implemented by a firststorage system designated for PHI storage, and the second memory can beimplemented by a second storage system designated for storage ofde-identified data. The first storage system can be protected fromaccess by the annotating system, while the second storage system can beaccessible by the annotating system. The medical picture archiveintegration system 2600 can be operable to perform the de-identificationfunction on data in first storage system to generate de-identified data.The de-identified data can then be stored in the second storage systemfor access by the annotating system. The first and second storagesystems can be physically separate, each utilizing at least one of theirown, separate memory devices. Alternatively, the first and secondstorage systems can be virtually separate, where data is stored inseparate virtual memory locations on the same set of memory devices.Firewalls, virtual machines, and/or other protected containerization canbe utilized to enforce the separation of data in each storage system, toprotect the first storage system from access by the annotating systemand/or from other unauthorized access, and/or to ensure that only dataof the first storage system that has been properly de-identified throughapplication of the de-identification function can be stored in thesecond storage system.

As shown in FIG. 8A, the medical picture archive system 2620 can receiveimage data from a plurality of modality machines 2622, such as CTmachines, Mill machines, x-ray machines, and/or other medical imagingmachines that produce medical scans. The medical picture archive system2620 can store this image data in a DICOM image format and/or can storethe image data in a plurality of medical scan entries 352 as describedin conjunction with some or all of the attributes described inconjunction with FIGS. 4A and 4B. While “DICOM image” will be usedherein to refer to medical scans stored by the medical picture archivesystem 2620, the medical picture archive integration system 2600 canprovide integration support for medical picture archive systems 2620that store medical scans in other formats.

The medical picture archive integration system 2600 can include areceiver 2602 and a transmitter 2604, operable to transmit and receivedata from the medical picture archive system 2620, respectively. Forexample, the receiver 2602 and transmitter 2604 can be configured toreceive and transmit data, respectively, in accordance with a DICOMcommunication protocol and/or another communication protocol recognizedby the medical image archive system 2620. The receiver can receive DICOMimages from the medical picture archive system 2620. The transmitter2604 can send annotated DICOM files to the medical picture archivesystem 2620.

DICOM images received via receiver 2602 can be sent directly to ade-identification system 2608. The de-identification system 2608 can beoperable to perform a de-identification function on the first DICOMimage to identify at least one patient identifier in the DICOM image,and to generate a de-identified medical scan that does not include theidentified at least one patient identifier. As used herein, a patientidentifier can include any patient identifying data in the image data,header, and/or metadata of a medical scan, such as a patient ID numberor other unique patient identifier, an accession number, aservice-object pair (SOP) instance unique identifier (UID) field, scandate and/or time that can be used to determine the identity of thepatient that was scanned at that date and/or time, and/or other privatedata corresponding to the patient, doctor, or hospital. In someembodiments, the de-identified medical scan is still in a DICOM imageformat. For example, a duplicate DICOM image that does not include thepatient identifiers can be generated, and/or the original DICOM imagecan be altered such that the patient identifiers of the new DICOM imageare masked, obfuscated, removed, replaced with a custom fiducial, and/orotherwise anonymized. In other embodiments, the de-identified medicalscan is formatted in accordance with a different image format and/ordifferent data format that does not include the identifying information.In some embodiments, other private information, for example, associatedwith a particular doctor or other medical professional, can beidentified and anonymized as well.

Some patient identifying information can be included in a DICOM headerof the DICOM image, for example, in designated fields for patientidentifiers. These corresponding fields can be anonymized within thecorresponding DICOM header field. Other patient identifying informationcan be included in the image itself, such as in medical scan image data410. For example, the image data can include a patient name or otheridentifier that was handwritten on a hard copy of the image before theimage was digitized. As another example, a hospital administered armbandor other visual patient information in the vicinity of the patient mayhave been captured in the image itself. A computer vision model candetect the presence of these identifiers for anonymization, for example,where a new DICOM image includes a fiducial image that covers theidentifying portion of the original DICOM image. In some embodiments,patient information identified in the DICOM header can be utilized todetect corresponding patient information in the image itself. Forexample, a patient name extracted from the DICOM header beforeanonymization can be used to search for the patient name in the imageand/or to detect a location of the image that includes the patient name.In some embodiments, the de-identification system 2608 is implemented bythe de-identification system discussed in conjunction with FIGS. 10A,10B and 11, and/or utilizes functions and/or operations discussed inconjunction with FIGS. 10A, 10B and 11.

The de-identified medical scan can be stored in de-identified imagestorage system 2610 and the annotating system 2612 can access thede-identified medical scan from the de-identified image storage system2610 for processing. The de-identified storage system can archive aplurality of de-identified DICOM images and/or can serve as temporarystorage for the de-identified medical scan until processing of thede-identified medical scan by the annotating system 2612 is complete.The annotating system 2612 can generate annotation data by performing aninference function on the de-identified medical scan, utilizing themodel parameters received from the central server system 2640. Theannotation data can correspond to some or all of the diagnosis data 440as discussed in conjunction with FIGS. 4A and 4B. In come embodiments,the annotating system 2612 can utilize the model parameters to performinference step 1354, the detection step 1372, the abnormalityclassification step 1374, the similar scan identification step 1376,and/or the display parameter step 1378 of the medical scan imageanalysis system 112, as discussed in conjunction with FIG. 7B, onde-identified medical scans received from the medical picture archivesystem 2620.

In some embodiments, model parameters for a plurality of inferencefunctions can be received from the central server system 2640, forexample, where each inference function corresponds to one of a set ofdifferent scan categories. Each scan category can correspond to a uniquecombination of one or a plurality of scan modalities, one of a pluralityof anatomical regions, and/or other scan classifier data 420. Forexample, a first inference function can be trained on and intended forde-identified medical scans corresponding chest CT scans, and a secondinference function can be trained on and intended for de-identifiedmedical scans corresponding to head MM scans. The annotating system canselect one of the set of inference functions based on determining thescan category of the DICOM image, indicated in the de-identified medicalscan, and selecting the inference function that corresponds to thedetermined scan category.

To ensure that scans received from the medical picture archive system2620 match the set of scan categories for which the annotating system isoperable to perform a corresponding inference function, the transmittercan transmit requests, such as DICOM queries, indicating image typeparameters such as parameters corresponding to scan classifier data 420,for example indicating one or more scan modalities, one or moreanatomical regions, and/or other parameters. For example, the requestcan indicate that all incoming scans that match the set of scancategories corresponding to a set of inference functions the annotatingsystem 2612 for which the annotating system has obtained modelparameters from the central server system 2640 and is operable toperform.

Once the annotation data is generated by performing the selectedinference function, the annotating system 2612 can generate an annotatedDICOM file for transmission to the medical image archive system 2620 forstorage. The annotated DICOM file can include some or all of the fieldsof the diagnosis data 440 and/or abnormality annotation data 442 ofFIGS. 4A and 4B. The annotated DICOM file can include scan overlay data,providing location data of an identified abnormality and/or display datathat can be used in conjunction with the original DICOM image toindicate the abnormality visually in the DICOM image and/or to otherwisevisually present the annotation data, for example, for use with themedical scan assisted review system 102. For example, a DICOMpresentation state file can be generated to indicate the location of anabnormality identified in the de-identified medical scan. The DICOMpresentation state file can include an identifier of the original DICOMimage, for example, in metadata of the DICOM presentation state file, tolink the annotation data to the original DICOM image. In otherembodiments, a full, duplicate DICOM image is generated that includesthe annotation data with an identifier linking this duplicate annotatedDICOM image to the original DICOM image.

The identifier linking the annotated DICOM file to the original DICOMimage can be extracted from the original DICOM file by thede-identification system 2608, thus enabling the medical picture archivesystem 2620 to link the annotated DICOM file to the original DICOM imagein its storage. For example, the de-identified medical scan can includean identifier that links the de-identified medical scan to the originalDICOM file, but does not link the de-identified medical scan to apatient identifier or other private data.

In some embodiments, generating the annotated DICOM file includesaltering one or more fields of the original DICOM header. For example,standardized header formatting function parameters can be received fromthe central server system and can be utilized by the annotating systemto alter the original DICOM header to match a standardized DICOM headerformat. The standardized header formatting function can be trained in asimilar fashion to other medical scan analysis functions discussedherein and/or can be characterized by some or all fields of a medicalscan analysis function entry 356. The annotating system can perform thestandardized header formatting function on a de-identified medical scanto generate a new, standardized DICOM header for the medical scan to besent back to the medical picture archive system 2620 in the annotatedDICOM file and/or to replace the header of the original DICOM file. Thestandardized header formatting function can be run in addition to otherinference functions utilized to generate annotation data. In otherembodiments, the medical picture archive integration system 2600 isimplemented primarily for header standardization for medical scansstored by the medical picture archive system 2620. In such embodiments,only the standardized header formatting function is performed on thede-identified data to generate a modified DICOM header for the originalDICOM image, but the de-identified medical scan is not annotated.

In some embodiments of header standardization, the annotation system canstore a set of acceptable, standardized entries for some or all of theDICOM header fields, and can select one of the set of acceptable,standardized entries in populating one or more fields of the new DICOMheader for the annotated DICOM file. For example, each of the set ofscan categories determined by the annotating system can correspond to astandardized entry of one or more fields of the DICOM header. The newDICOM header can thus be populated based on the determined scancategory.

In some embodiments, each of the set of standardized entries can bemapped to a set of related, non-standardized entries, such as entries ina different order, commonly misspelled entries, or other similar entriesthat do not follow a standardized format. For example, one of the set ofacceptable, standardized entries for a field corresponding to a scancategory can include “Chest CT”, which can be mapped to a set ofsimilar, non-standardized entries which can include “CT chest”,“computerized topography CT”, and/or other entries that are notstandardized. In such embodiments, the annotating system can determinethe original DICOM header is one of the similar non-standardizedentries, and can select the mapped, standardized entry as the entry forthe modified DICOM header. In other embodiments, the image data itselfand/or or other header data can be utilized by the annotation system todetermine a standardized field. For example, an input quality assurancefunction 1106 can be trained by the central server system and sent tothe annotating system to determine one or more appropriate scanclassifier fields, or one or more other DICOM header fields, based onthe image data or other data of the de-identified medical scan. One ormore standardized labels can be assigned to corresponding fields of themodified DICOM header based on the one or more fields determined by theinput quality assurance function.

In some embodiments, the DICOM header is modified based on theannotation data generated in performing the inference function. Inparticular, a DICOM priority header field can be generated and/ormodified automatically based on the severity and/or time-sensitivity ofthe abnormalities detected in performing the inference function. Forexample, a DICOM priority header field can be changed from a lowpriority to a high priority in response to annotation data indicating abrain bleed in the de-identified medical scan of a DICOM imagecorresponding to a head CT scan, and a new DICOM header that includesthe high priority DICOM priority header field can be sent back to themedical picture archive system 2620 to replace or otherwise be mapped tothe original DICOM image of the head CT scan.

In various embodiments, the medical picture archive system 2620 isdisconnected from network 2630, for example, to comply with requirementsregarding Protected Health Information (PHI), such as patientidentifiers and other private patient information included in the DICOMimages and/or otherwise stored by the medical picture archive system2620. The medical picture archive integration system 2600 can enableprocessing of DICOM images while still protecting private patientinformation by first de-identifying DICOM data by utilizingde-identification system 2608. The de-identification system 2608 canutilize designated processors and memory of the medical picture archiveintegration system, for example, designated for PHI. Thede-identification system 2608 can be decoupled from the network 2630 toprevent the DICOM images that still include patient identifiers frombeing accessed via the network 2630. For example, as shown in FIG. 8A,the de-identification system 2608 is not connected to network interface2606. Furthermore, only the de-identification system 2608 has access tothe original DICOM files received from the medical picture archivesystem 2620 via receiver 2602. The de-identified image storage system2610 and annotating system 2612, as they are connected to network 2630via network interface 2606, only store and have access to thede-identified medical scan produced by the de-identification system2608.

This containerization that separates the de-identification system 2608from the de-identified image storage system 2610 and the annotatingsystem 2612 is further illustrated in FIG. 8B, which presents anembodiment of the medical picture archive integration system 2600. Thede-identification system 2608 can include its own designated memory 2654and processing system 2652, connected to receiver 2602 via bus 2659. Forexample, this memory 2654 and processing system 2652 can be designatedfor PHI, and can adhere to requirements for handling PHI. The memory2654 can store executable instructions that, when executed by theprocessing system 2652, enable the de-identification system to performthe de-identification function on DICOM images received via receiver2602 of the de-identification system. The incoming DICOM images can betemporarily stored in memory 2654 for processing, and patientidentifiers detected in performing the de-identification function can betemporarily stored in memory 2654 to undergo anonymization. Interface2655 can transmit the de-identified medical scan to interface 2661 foruse by the de-identified image storage system 2610 and the annotatingsystem 2612. Interface 2655 can be protected from transmitting originalDICOM files and can be designated for transmission of de-identifiedmedical scan only.

Bus 2669 connects interface 2661, as well as transmitter 2604 andnetwork interface 2606, to the de-identified image storage system 2610and the annotating system 2612. The de-identified image storage system2610 and annotating system 2612 can utilize separate processors andmemory, or can utilize shared processors and/or memory. For example, thede-identified image storage system 2610 can serve as temporary memory ofthe annotating system 2612 as de-identified images are received andprocessed to generate annotation data.

As depicted in FIG. 8B, the de-identified image storage system 2610 caninclude memory 2674 that can temporarily store incoming de-identifiedmedical scans as it undergoes processing by the annotating system 2612and/or can archive a plurality of de-identified medical scanscorresponding to a plurality of DICOM images received by the medicalpicture archive integration system 2600. The annotating system 2612 caninclude a memory 2684 that stores executable instructions that, whenexecuted by processing system 2682, cause the annotating system 2612perform a first inference function on de-identified medical scan togenerate annotation data by utilizing the model parameters received viainterface 2606, and to generate an annotated DICOM file based on theannotation data for transmission via transmitter 2604. The modelparameters can be stored in memory 2684, and can include modelparameters for a plurality of inference functions, for example,corresponding to a set of different scan categories.

The medical picture archive integration system can be an onsite system,installed at a first geographic site, such as a hospital or othermedical entity that is affiliated with the medical picture archivesystem 2620. The hospital or other medical entity can further beresponsible for the PHI of the de-identification system, for example,where the memory 2654 and processing system 2652 are owned by,maintained by, and/or otherwise affiliated with the hospital or othermedical entity. The central server system 2640 can be located at asecond, separate geographic site that is not affiliated with thehospital or other medical entity and/or at a separate geographic sitethat is not affiliated with the medical picture archive system 2620. Thecentral server system 2640 can be a server configured to be outside thenetwork firewall and/or out outside the physical security of thehospital or other medical entity or otherwise not covered by theparticular administrative, physical and technical safeguards of thehospital or other medical entity.

FIG. 8C further illustrates how model parameters can be updated overtime to improve existing inference functions and/or to add new inferencefunctions, for example corresponding to new scan categories. Inparticular, the some or all of the de-identified medical scans generatedby the de-identification system 2608 can be transmitted back to thecentral server system, and the central server system 2640 can train onthis data to improve existing models by producing updated modelparameters of an existing inference function and/or to generate newmodels, for example, corresponding to new scan categories, by producingnew model parameters for new inference functions. For example, thecentral server system 2640 can produce updated and/or new modelparameters by performing the training step 1352 of the medical scanimage analysis system 112, as discussed in conjunction with FIG. 7A, ona plurality of de-identified medical scans received from the medicalpicture archive integration system 2600.

The image type parameters can be determined by the central server systemto dictate characteristics of the set of de-identified medical scans tobe received to train and/or retrain the model. For example, the imagetype parameters can correspond to one or more scan categories, canindicate scan classifier data 420, can indicate one or more scanmodalities, one or more anatomical regions, a date range, and/or otherparameters. The image type parameters can be determined by the centralserver system based on training parameters 620 determined for thecorresponding inference function to be trained, and/or based oncharacteristics of a new and/or existing scan category corresponding tothe inference function to be trained. The image type parameters can besent to the medical picture archive integration system 2600, and arequest such as a DICOM query can be sent to the medical picture archivesystem 2620, via transmitter 2604, that indicates the image typeparameters. For example, the processing system 2682 can be utilized togenerate the DICOM query based on the image type parameters receivedfrom the central server system 2640. The medical picture archive systemcan automatically transmit one or more DICOM images to the medicalpicture archive integration system in response to determining that theone or more DICOM images compares favorably to the image typeparameters. The DICOM images received in response can be de-identifiedby the de-identification system 2608. In some embodiments, thede-identified medical scans can be transmitted directly to the centralserver system 2640, for example, without generating annotation data.

The central server system can generate the new and/or updated modelparameters by training on the received set of de-identified medicalscans, and can transmit the new and/or updated model parameters to thede-identified storage system. If the model parameters correspond to anew inference function for a new scan category, the medical picturearchive integration system 2600 can generate a request, such as a DICOMquery, for transmission to the medical picture archive system indicatingthat incoming scans corresponding to image type parameters correspondingto the new scan category be sent to the medical picture archiveintegration system. The annotating system can update the set ofinference functions to include the new inference function, and theannotating system can select the new inference function from the set ofinference functions for subsequently generated de-identified medicalscans by the de-identification system by determining each of thesede-identified medical scans indicate the corresponding DICOM imagecorresponds to the new scan category. The new model parameters can beutilized to perform the new inference function on each of thesede-identified medical scans to generate corresponding annotation data,and an annotated DICOM file corresponding to each of these de-identifiedmedical scans can be generated for transmission to the medical picturearchive system via the transmitter.

In some embodiments, the central server system 2640 receives a pluralityof de-identified medical scans from a plurality of medical picturearchive integration system 2600, for example, each installed at aplurality of different hospitals or other medical entities, via thenetwork 2630. The central server system can generate training sets byintegrating de-identified medical scans from some or all of theplurality of medical picture archive integration systems 2600 to trainone or more inference functions and generate model parameters. Theplurality of medical picture archive integration systems 2600 canutilize the same set of inference functions or different sets ofinference functions. In some embodiments, the set of inference functionsutilized by the each of the plurality of medical picture archive systems2620 are trained on different sets of training data. For example, thedifferent sets of training data can correspond to the set ofde-identified medical scans received from the corresponding medicalpicture archive integration system 2600.

In some embodiments, the medical scan diagnosing system 108 can beutilized to implement the annotating system 2612, where thecorresponding subsystem processing device 235 and subsystem memorydevice 245 of the medical scan diagnosing system 108 are utilized toimplement the processing system 2682 and the memory 2684, respectively.Rather than receiving the medical scans via the network 150 as discussedin conjunction with FIG. 6A, the medical scan diagnosing system 108 canperform a selected medical scan inference function 1105 on an incomingde-identified medical scan generated by the de-identification system2608 and/or retrieved from the de-identified image storage system 2610.Memory 2684 can store the set of medical scan inference functions 1105,each corresponding to a scan category 1120, where the inference functionis selected from the set based on determining the scan category of thede-identified medical scan and selecting the corresponding inferencefunction. The processing system 2682 can perform the selected inferencefunction 1105 to generate the inference data 1110, which can be furtherutilized by the annotating system 2612 to generate the annotated DICOMfile for transmission back to the medical picture archive system 2620.New medical scan inference functions 1105 can be added to the set whencorresponding model parameters are received from the central serversystem. The remediation step 1140 can be performed locally by theannotating system 2612 and/or can be performed by the central serversystem 2640 by utilizing one or more de-identified medical scans andcorresponding annotation data sent to the central server system 2640.Updated model parameters can be generated by the central server system2640 and sent to the medical picture archive integration system 2600 asa result of performing the remediation step 1140.

The central server system 2640 can be implemented by utilizing one ormore of the medical scan subsystems 101, such as the medical scan imageanalysis system 112 and/or the medical scan diagnosing system 108, toproduce model parameters for one or more inference functions. Thecentral server system can store or otherwise communicate with a medicalscan database 342 that includes the de-identified medical scans and/orannotation data received from one or more medical picture archiveintegration systems 2600. Some or all entries of the medical scandatabase 342 can be utilized to as training data to produce modelparameters for one or more inference functions. These entries of themedical scan database 342 can be utilized by other subsystems 101 asdiscussed herein. For example, other subsystems 101 can utilize thecentral server system 2640 to fetch medical scans and/or correspondingannotation data that meet specified criteria. The central server system2640 can query the medical picture archive integration system 2600 basedon this criteria, and can receive de-identified medical scans and/orannotation data in response. This can be sent to the requestingsubsystem 101 directly and/or can be added to the medical scan database342 or another database of the database storage system 140 for access bythe requesting subsystem 101.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a user database 344 storing user profileentries corresponding to each of a plurality of medical entities thateach utilize a corresponding one of a plurality of medical picturearchive integration systems 2600. For example, basic user datacorresponding to the medical entity can be stored as basic user data, anumber of scans or other consumption information indicating usage of oneor more inference functions by corresponding medical picture archiveintegration system can be stored as consumption usage data, and/or anumber of scans or other contribution information indicatingde-identified scans sent to the central server system as training datacan be stored as contribution usage data. The user profile entry canalso include inference function data, for example, with a list of modelparameters or function identifiers, such as medical scan analysisfunction identifiers 357, of inference functions currently utilized bythe corresponding medical picture archive integration system 2600. Theseentries of the user database 344 can be utilized by other subsystems 101as discussed herein.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a medical scan analysis function database346 to store model parameters, training data, or other information forone or more inference functions as medical scan analysis functionentries 356. In some embodiments, model parameter data 623 can indicatethe model parameters and function classifier data 610 can indicate thescan category of inference function entries. In some embodiments, themedical scan analysis function entry 356 can further include usageidentifying information indicating a medical picture archive integrationsystem identifier, medical entity identifier, and/or otherwiseindicating which medical archive integration systems and/or medicalentities have received the corresponding model parameters to utilize theinference function corresponding to the medical scan analysis functionentry 356. These entries of the medical scan analysis function database346 can be utilized by other subsystems 101 as discussed herein.

In some embodiments, the de-identification function is a medical scananalysis function, for example, with a corresponding medical scananalysis function entry 356 in the medical scan analysis functiondatabase 346. In some embodiments, the de-identification function istrained by the central server system 2640. For example, the centralserver system 2640 can send de-identification function parameters to themedical picture archive integration system 2600 for use by thede-identification system 2608. In embodiments with a plurality ofmedical picture archive integration systems 2600, each of the pluralityof medical picture archive integration systems 2600 can utilize the sameor different de-identification functions. In some embodiments, thede-identification function utilized by the each of the plurality ofmedical picture archive integration systems 2600 are trained ondifferent sets of training data. For example, the different sets oftraining data can correspond to each different set of de-identifiedmedical scans received from each corresponding medical picture archiveintegration system 2600.

In some embodiments, as illustrated in FIGS. 8D-8F, the medical picturearchive integration system 2600 can further communicate with a reportdatabase 2625, such as a Radiology Information System (RIS), thatincludes a plurality of medical reports corresponding to the DICOMimages stored by the medical picture archive system 2620.

As shown in FIG. 8D, the medical picture archive integration system 2600can further include a receiver 2603 that receives report data,corresponding to the DICOM image, from report database 2625. The reportdatabase 2625 can be affiliated with the medical picture archive system2620 and can store report data corresponding to DICOM images stored inthe medical picture archive system. The report data of report database2625 can include PHI, and the report database 2625 can thus bedisconnected from network 2630.

The report data can include natural language text, for example,generated by a radiologist that reviewed the corresponding DICOM image.The report data can be used to generate the de-identified medical scan,for example, where the de-identification system 2608 performs a naturallanguage analysis function on the report data to identify patientidentifying text in the report data. The de-identification system 2608can utilize this patient identifying text to detect matching patientidentifiers in the DICOM image to identify the patient identifiers ofthe DICOM image and generate the de-identified medical scan. In someembodiments, the report data can be de-identified by obfuscating,hashing, removing, replacing with a fiducial, or otherwise anonymizingthe identified patient identifying text to generate de-identified reportdata.

The de-identified report data can be utilized by the annotating system2612, for example, in conjunction with the DICOM image, to generate theannotation data. For example, the annotating system 2612 can perform anatural language analysis function on the de-identified natural languagetext of the report data to generate some or all of the annotation data.In some embodiments, the de-identified report data is sent to thecentral server system, for example, to be used as training data forinference functions, for natural language analysis functions, for othermedical scan analysis functions, and/or for use by at least one othersubsystem 101. For example, other subsystems 101 can utilize the centralserver system 2640 to fetch medical reports that correspond toparticular medical scans or otherwise meet specified criteria. Thecentral server system 2640 can query the medical picture archiveintegration system 2600 based on this criteria, and can receivede-identified medical reports in response. This can be sent to therequesting subsystem 101 directly, can be added to the medical scandatabase 342, a de-identified report database, or another database ofthe database storage system 140 for access by the requesting subsystem101.

In some embodiments the medical picture archive integration system 2600can query the report database 2625 for the report data corresponding toa received DICOM image by utilizing a common identifier extracted fromthe DICOM image.

In some embodiments, the report data can correspond to a plurality ofDICOM images. For example, the report data can include natural languagetext describing a plurality of medical scans of a patient that caninclude multiple sequences, multiple modalities, and/or multiple medicalscans taken over time. In such embodiments, the patient identifying textand/or annotation data detected in the report data can also be appliedto de-identify and/or generate annotation data for the plurality ofDICOM images it describes. In such embodiments, the medical picturearchive integration system 2600 can query the medical picture archivesystem 2620 for one or more additional DICOM images corresponding to thereport data, and de-identified data and annotation data for theseadditional DICOM images can be generated accordingly by utilizing thereport data.

In some embodiments, as shown in FIG. 8E, the medical picture archivesystem 2620 communicates with the report database 2625. The medicalpicture archive system 2620 can request the report data corresponding tothe DICOM image from the report database 2625, and can transmit thereport data to the medical picture archive integration system 2600 via aDICOM communication protocol for receipt via receiver 2602. The medicalpicture archive system 2620 can query the report database 2625 for thereport data, utilizing a common identifier extracted from thecorresponding DICOM image, in response to determining to send thecorresponding DICOM image to the medical picture archive integrationsystem 2600.

FIG. 8F presents an embodiment where report data is generated by theannotating system 2612 and is transmitted, via a transmitter 2605, tothe report database 2625, for example via a DICOM communication protocolor other protocol recognized by the report database 2625. In otherembodiments, the report data is instead transmitted via transmitter 2604to the medical picture archive system 2620, and the medical picturearchive system 2620 transmits the report data to the report database2625.

The report data can be generated by the annotating system 2612 as outputof performing the inference function on the de-identified medical scan.The report data can include natural language text data 448 generatedautomatically based on other diagnosis data 440 such as abnormalityannotation data 442 determined by performing the inference function, forexample, by utilizing a medical scan natural language generatingfunction trained by the medical scan natural language analysis system114. The report data can be generated instead of, or in addition to, theannotated DICOM file.

FIG. 9 presents a flowchart illustrating a method for execution by amedical picture archive integration system 2600 that includes a firstmemory and a second memory that store executable instructions that, whenexecuted by at least one first processor and at least one secondprocessor, respectfully, cause the medical picture archive integrationsystem to perform the steps below. In various embodiments, the firstmemory and at least one first processor are implemented by utilizing,respectfully, the memory 2654 and processing system 2652 of FIG. 8B. Invarious embodiments, the second memory is implemented by utilizing thememory 2674 and/or the memory 2684 of FIG. 8B. In various embodiments,the at least one second processor is implemented by utilizing theprocessing system 2682 of FIG. 8B.

Step 2702 includes receiving, from a medical picture archive system viaa receiver, a first DICOM image for storage in the first memory,designated for PHI, where the first DICOM image includes at least onepatient identifier. Step 2704 includes performing, via at least onefirst processor coupled to the first memory and designated for PHI, ade-identification function on the first DICOM image to identify the atleast one patient identifier and generate a first de-identified medicalscan that does not include the at least one patient identifier.

Step 2706 includes storing the first de-identified medical scan in asecond memory that is separate from the first memory. Step 2708 includesreceiving, via a network interface communicating with a network thatdoes not include the medical picture archive system, first modelparameters from a central server.

Step 2710 includes retrieving the first de-identified medical scan fromthe second memory. Step 2712 includes utilizing the first modelparameters to perform a first inference function on the firstde-identified medical scan to generate first annotation data via atleast one second processor that is different from the at least one firstprocessor. Step 2714 includes generating, via the at least one secondprocessor, a first annotated DICOM file for transmission to the medicalpicture archive system via a transmitter, where the first annotatedDICOM file includes the first annotation data and further includes anidentifier that indicates the first DICOM image. In various embodiments,the first annotated DICOM file is a DICOM presentation state file.

In various embodiments, the second memory further includes operationalinstructions that, when executed by the at least one second processor,further cause the medical picture archive integration system to retrievea second de-identified medical scan from the de-identified image storagesystem, where the second de-identified medical scan was generated by theat least one first processor by performing the de-identificationfunction on a second DICOM image received from the medical picturearchive system. The updated model parameters are utilized to perform thefirst inference function on the second de-identified medical scan togenerate second annotation data. A second annotated DICOM file isgenerated for transmission to the medical picture archive system via thetransmitter, where the second annotated DICOM file includes the secondannotation data and further includes an identifier that indicates thesecond DICOM image.

In various embodiments, the second memory stores a plurality ofde-identified medical scans generated by the at least one firstprocessor by performing the de-identification function on acorresponding plurality of DICOM images received from the medicalpicture archive system via the receiver. The plurality of de-identifiedmedical scans is transmitted to the central server via the networkinterface, and the central server generates the first model parametersby performing a training function on training data that includes theplurality of de-identified medical scans.

In various embodiments, the central server generates the first modelparameters by performing a training function on training data thatincludes a plurality of de-identified medical scans received from aplurality of medical picture archive integration systems via thenetwork. Each of the plurality of medical picture archive integrationsystems communicates bidirectionally with a corresponding one of aplurality of medical picture archive systems, and the plurality ofde-identified medical scans corresponds to a plurality of DICOM imagesstored by the plurality of medical picture archive integration systems.

In various embodiments, the first de-identified medical scan indicates ascan category of the first DICOM image. The second memory further storesoperational instructions that, when executed by the at least one secondprocessor, further cause the medical picture archive integration systemto select the first inference function from a set of inference functionsbased on the scan category. The set of inference functions correspondsto a set of unique scan categories that includes the scan category. Invarious embodiments, each unique scan category of the set of unique scancategories is characterized by one of a plurality of modalities and oneof a plurality of anatomical regions.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,further cause the medical picture archive integration system to receivea plurality of DICOM image data from the medical picture archive systemvia the receiver for storage in the first memory in response to a querytransmitted to the medical picture archive system via the transmitter.The query is generated by the medical picture archive integration systemin response to a request indicating a new scan category received fromthe central server via the network. The new scan category is notincluded in the set of unique scan categories, and the plurality ofDICOM image data corresponds to the new scan category. Thede-identification function is performed on the plurality of DICOM imagedata to generate a plurality of de-identified medical scans fortransmission to the central server via the network.

The second memory further stores operational instructions that, whenexecuted by the at least one second processor, further cause the medicalpicture archive integration system to receive second model parametersfrom the central server via the network for a new inference functioncorresponding to the new scan category. The set of inference functionsis updated to include the new inference function. The secondde-identified medical scan is retrieved from the first memory, where thesecond de-identified medical scan was generated by the at least onefirst processor by performing the de-identification function on a secondDICOM image received from the medical picture archive system. The newinference function is selected from the set of inference functions bydetermining the second de-identified medical scan indicates the secondDICOM image corresponds to the new scan category. The second modelparameters are utilized to perform the new inference function on thesecond de-identified medical scan to generate second annotation data. Asecond annotated DICOM file is generated for transmission to the medicalpicture archive system via the transmitter, where the second annotatedDICOM file includes the second annotation data and further includes anidentifier that indicates the second DICOM image.

In various embodiments, the medical picture archive integration systemgenerates parameter data for transmission to the medical picture archivesystem that indicates the set of unique scan categories. The medicalpicture archive system automatically transmits the first DICOM image tothe medical picture archive integration system in response todetermining that the first DICOM image compares favorably to one of theset of unique scan categories.

In various embodiments, the second memory further stores operationalinstructions that, when executed by the at least one second processor,cause the medical picture archive integration system to generate anatural language report data is based on the first annotation data andto transmit, via a second transmitter, the natural language report datato a report database associated with the medical picture archiveintegration system, where the natural language report data includes anidentifier corresponding to the first DICOM image.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,cause the medical picture archive integration system to receive, via asecond receiver, a natural language report corresponding to the firstDICOM image from the report database. A set of patient identifying textincluded in the natural language report are identified. Performing thede-identification function on the first DICOM image includes searchingthe first DICOM image for the set of patient identifying text toidentify the at least one patient identifier.

In various embodiments, the first memory is managed by a medical entityassociated with the medical picture archive system. The medical picturearchive integration system is located at a first geographic sitecorresponding to the medical entity, and the central server is locatedat a second geographic site. In various embodiments, the first memory isdecoupled from the network to prevent the first DICOM image thatincludes the at least one patient identifier from being communicated viathe network. In various embodiments, the medical picture archive systemis a Picture Archive and Communication System (PACS) server, and thefirst DICOM image is received in response to a query sent to the medicalpicture archive system by the transmitter in accordance with a DICOMcommunication protocol.

FIG. 10A presents an embodiment of a de-identification system 2800. Thede-identification system 2800 can be utilized to implement thede-identification system 2608 of FIGS. 8A-8F. In some embodiments, thede-identification system 2800 can be utilized by other subsystems tode-identify image data, medical report data, private fields of medicalscan entries 352 such as patient identifier data 431, and/or otherprivate fields stored in databases of the database memory device 340.

The de-identification system can be operable to receive, from at leastone first entity, a medical scan and a medical report corresponding tothe medical scan. A set of patient identifiers can be identified in asubset of fields of a header of the medical scan. A header anonymizationfunction can be performed on each of the set of patient identifiers togenerate a corresponding set of anonymized fields. A de-identifiedmedical scan can be generated by replacing the subset of fields of theheader of the medical scan with the corresponding set of anonymizedfields.

A subset of patient identifiers of the set of patient identifiers can beidentified in the medical report by searching text of the medical reportfor the set of patient identifiers. A text anonymization function can beperformed on the subset of patient identifiers to generate correspondinganonymized placeholder text for each of the subset of patientidentifiers. A de-identified medical report can be generated byreplacing each of the subset of patient identifiers with thecorresponding anonymized placeholder text. The de-identified medicalscan and the de-identified medical report can be transmitted to a secondentity via a network.

As shown in FIG. 10A, the de-identification system 2800 can include atleast one receiver 2802 operable to receive medical scans, such asmedical scans in a DICOM image format. The at least one receiver 2802 isfurther operable to receive medical reports, such as report data 449 orother reports containing natural language text diagnosing, describing,or otherwise associated the medical scans received by thede-identification system. The medical scans and report data can bereceived from the same or different entity, and can be received by thesame or different receiver 2802 in accordance with the same or differentcommunication protocol. For example, the medical scans can be receivedfrom the medical picture archive system 2620 of FIGS. 8A-8F and thereport data can be received from the report database 2625 of FIGS.8D-8F. In such embodiments, the receiver 2802 can be utilized toimplement the receiver 2602 of FIG. 8B.

The de-identification system 2800 can further include a processingsystem 2804 that includes at least one processor, and a memory 2806. Thememory 2806 can store operational instructions that, when executed bythe processing system, cause the de-identification system to perform atleast one patient identifier detection function on the received medicalscan and/or the medical report to identify a set of patient identifiersin the medical scan and/or the medical report. The operationalinstructions, when executed by the processing system, can further causethe de-identification system to perform an anonymization function on themedical scan and/or the medical report to generate a de-identifiedmedical scan and/or a de-identified medical report that do not includethe set of patient identifiers found in performing the at least onepatient identifier detection function. Generating the de-identifiedmedical scan can include generating a de-identified header andgenerating de-identified image data, where the de-identified medicalscan includes both the de-identified header and the de-identified imagedata. The memory 2806 can be isolated from Internet connectivity, andcan be designated for PHI.

The de-identification system 2800 can further include at least onetransmitter 2808, operable to transmit the de-identified medical scanand de-identified medical report. The de-identified medical scan andde-identified medical report can be transmitted back to the same entityfrom which they were received, respectively, and/or can be transmittedto a separate entity. For example, the at least one transmitter cantransmit the de-identified medical scan to the de-identified imagestorage system 2610 of FIGS. 8A-8F and/or can transmit the de-identifiedmedical scan to central server system 2640 via network 2630 of FIGS.8A-8F. In such embodiments, the transmitter 2808 can be utilized toimplement the interface 2655 of FIG. 8B. The receiver 2802, processingsystem 2804, memory 2806, and/or transmitter 2808 can be connected viabus 2810.

Some or all of the at least one patient identifier detection functionand/or at least one anonymization function as discussed herein can betrained and/or implemented by one or subsystems 101 in the same fashionas other medical scan analysis functions discussed herein, can be storedin medical scan analysis function database 346 of FIG. 3, and/or canotherwise be characterized by some or all fields of a medical scananalysis function entry 356 of FIG. 5.

The de-identification system 2800 can perform separate patientidentifier detection functions on the header of a medical report and/ormedical scan, on the text data of the medical report, and/or on theimage data of the medical scan, such as text extracted from the imagedata of the medical scan. Performance of each of these functionsgenerates an output of its own set of identified patient identifiers.Combining these sets of patient identifiers yields a blacklist term set.A second pass of the header of a medical report and/or medical scan, onthe text data of the medical report, and/or on the image data of themedical scan that utilizes this blacklist term set can catch any termsthat were missed by the respective patient identifier detectionfunction, and thus, the outputs of these multiple identificationprocesses can support each other. For example, some of the data in theheaders will be in a structured form and can thus be easier to reliablyidentify. This can be exploited and used to further anonymize theseidentifiers when they appear in free text header fields, report data,and/or in the image data of the medical scan. Meanwhile, unstructuredtext in free text header fields, report data, and/or image data of themedical scan likely includes pertinent clinical information to bepreserved in the anonymization process, for example, so it can beleveraged by at least one subsystem 101 and/or so it can be leveraged intraining at least one medical scan analysis function.

At least one first patient identifier detection function can includeextracting the data in a subset of fields of a DICOM header, or anotherheader or other metadata of the medical scan and/or medical report witha known type that corresponds to patient identifying data. For example,this patient identifying subset of fields can include a name field, apatient ID number field or other unique patient identifier field, a datefield, a time field, an age field, an accession number field, SOPinstance UID, and/or other fields that could be utilized to identify thepatient and/or contain private information. A non-identifying subset offields of the header can include hospital identifiers, machine modelidentifiers, and/or some or all fields of medical scan entry 352 that donot correspond to patient identifying data. The patient identifyingsubset of fields and the non-identifying subset of fields can bemutually exclusive and collectively exhaustive with respect to theheader. The at least one patient identifier function can includegenerating a first set of patient identifiers by ignoring thenon-identifying subset of fields and extracting the entries of thepatient identifying subset of fields only. This first set of patientidentifiers can be anonymized to generate a de-identified header asdiscussed herein.

In some embodiments, at least one second patient identifier detectionfunction can be performed on the report data of the medical report. Theat least one second patient identifier detection function can includeidentifying patient identifying text in the report data by performing anatural language analysis function, for example, trained by the medicalscan natural language analysis system 114. For example, the at least onesecond patient identifier detection function can leverage the knownstructure of the medical report and/or context of the medical report. Asecond set of patient identifiers corresponding to the patientidentifying text can be determined, and the second set of patientidentifiers can be anonymized to generate a de-identified medicalreport. In some embodiments, a de-identified medical report includesclinical information, for example, because the portion of the originalmedical report that includes the clinical information was deemed to befree of patient identifying text and/or because the portion of theoriginal medical report that includes the clinical information wasdetermined to include pertinent information to be preserved.

In some embodiments, the medical report includes image datacorresponding to freehand or typed text. For example the medical reportcan correspond to a digitized scan of original freehand text written bya radiologist or other medical professional. In such embodiments, thepatient identifier detection function can first extract the text fromthe freehand text in the image data to generate text data before the atleast one second patient identifier detection function is performed onthe text of the medical report to generate the second set of patientidentifiers.

In some embodiments, the at least one second patient identifierdetection function can similarly be utilized to identify patientidentifying text in free text fields and/or unstructured text fields ofa DICOM header and/or other metadata of the medical scan and/or medicalreport data by performing a natural language analysis function, forexample, trained by the medical scan natural language analysis system114. A third set of patient identifiers corresponding to this patientidentifying text of the free text and/or unstructured header fields canbe determined, and the third set of patient identifiers can beanonymized to generate de-identified free text header field and/orunstructured header fields. In some embodiments, a de-identified freetext header field and/or unstructured header field includes clinicalinformation, for example, because the portion of the originalcorresponding header field that includes the clinical information wasdeemed to be free of patient identifying text and/or because the portionof the original corresponding header field that includes the clinicalinformation was determined to include pertinent information to bepreserved.

Patient identifiers can also be included in the image data of themedical scan itself. For example, freehand text corresponding to apatient name written on a hard copy of the medical scan beforedigitizing can be included in the image data, as discussed inconjunction with FIG. 10B. Other patient identifiers, such asinformation included on a patient wristband or other identifyinginformation located on or within the vicinity of the patient may havebeen captured when the medical scan was taken, and can thus be includedin the image. At least one third patient identifier detection functioncan include extracting text from the image data and/or detectingnon-text identifiers in the image data by performing a medical scanimage analysis function, for example, trained by the medical scan imageanalysis system 112. For example, detected text that corresponds to animage location known to include patient identifiers, detected text thatcorresponds to a format of a patient identifier, and/or or detected textor other image data determined to correspond to a patient identifier canbe identified. The at least one third patient identifier detectionfunction can further include identifying patient identifying text in thetext extracted from the image data by performing the at least one secondpatient identifier detection function and/or by performing a naturallanguage analysis function. A fourth set of patient identifierscorresponding to patient identifying text or other patient identifiersdetected in the image data of the medical scan can be determined, andthe fourth set of patient identifiers can be anonymized in the imagedata to generate de-identified image data of the medical scan asdescribed herein. In particular, the fourth set of patient identifierscan be detected in a set of regions of image data of the medical scan,and the set of regions of the image data can be anonymized.

In some embodiments, only a subset of the patient identifier detectionfunctions described herein are performed to generate respective sets ofpatient identifiers for anonymization. In some embodiments, additionalpatient identifier detection functions can be performed on the medicalscan and/or medical report to determine additional respective sets ofpatient identifiers for anonymization. The sets of patient identifiersoutputted by performing each patient identifier detection function canhave a null or non-null intersection. The sets of patient identifiersoutputted by performing each patient identifier function can have nullor non-null set differences.

Cases where the sets of patient identifiers have non-null setdifferences can indicate that a patient identifier detected by onefunction may have been missed by another function. The combined set ofpatient identifiers, for example, generated as the union of the sets ofsets of patient identifiers outputted by performing each patientidentifier function, can be used to build a blacklist term set, forexample, stored in memory 2806. The blacklist term set can designate thefinal set of terms to be anonymized. A second pass of header data,medical scans, medical reports, and/or any free text extracted from theheader data, the medical scan, and/or the medical report can beperformed by utilizing the blacklist term set to flag terms foranonymization that were not caught in performing the respective at leastone patient identifier detection function. For example, performing thesecond pass can include identifying at least one patient identifier ofthe blacklist term set in the header, medical report, and/or image dataof the medical scan. This can include by searching correspondingextracted text of the header, medical report, and/or image data forterms included in blacklist term set and/or by determining if each termin the extracted text is included in the blacklist term set.

In some embodiments, at least one patient identifier is not detecteduntil the second pass is performed. Consider an example where a freetext field of a DICOM header included a patient name that was notdetected in performing a respective patient identifier detectionfunction on the free text field of the DICOM header. However, thepatient name was successfully identified in the text of the medicalreport in performing a patient identifier detection function on themedical report. This patient name is added to the blacklist term list,and is detected in a second pass of the free text field of the DICOMheader. In response to detection in the second pass, the patient name ofthe free text field of the DICOM header can be anonymized accordingly togenerate a de-identified free text field. Consider a further examplewhere the patient name is included in the image data of the medicalscan, but was not detected in performing a respective patient identifierdetection function on the free text field of the DICOM header. In thesecond pass, this patient name can be detected in at least one region ofimage data of the medical scan by searching the image data for theblacklist term set.

In some embodiments, performing some or all of the patient identifierdetection functions includes identifying a set of non-identifying terms,such as the non-identifying subset of fields of the header. Inparticular, the non-identifying terms can include terms identified asclinical information and/or other terms determined to be preserved. Thecombined set of non-identifying terms, for example, generated as theunion of the sets of sets of non-identifying outputted by performingeach patient identifier function, can be used to build a whitelist termset, for example, stored in memory 2806. Performing the second pass canfurther include identifying at least one non-identifying term of thewhitelist term set in the header, medical report, and/or image data ofthe medical scan, and determining not to anonymize, or to otherwiseignore, the non-identifying term.

In various embodiments, some or all terms of the whitelist term set canbe removed from the blacklist term set. In particular, at least one termpreviously identified as a patient identifier in performing one or morepatient identifier detection functions is determined to be ignored andnot anonymized in response to determining the term is included in thewhitelist term set. This can help ensure that clinically importantinformation is not anonymized, and is thus preserved in thede-identified medical scan and de-identified medical report.

In some embodiments, the second pass can be performed after each of thepatient identifier detection functions are performed. For example,performing the anonymization function can include performing this secondpass by utilizing the blacklist term set to determine the final set ofterms to be anonymized. New portions of text in header fields, notpreviously detected in generating the first set of patient identifiersor the third set of patient identifiers, can be flagged foranonymization by determining these new portions of text correspond toterms of the blacklist term set. New portions of text the medicalreport, not previously detected in generating in the second set ofpatient identifiers, can be flagged for anonymization by determiningthese new portions of text correspond to terms of the blacklist termset. New regions of the image data of the medical scan, not previouslydetected in generating the fourth set of patient identifiers, can beflagged for anonymization by determining these new portions of textcorrespond to terms of the blacklist term set.

In some embodiments, the blacklist term set is built as each patientidentifier detection function is performed, and performance ofsubsequent patient identifier detection functions includes utilizing thecurrent blacklist term set. For example, performing the second patientidentifier detection function can include identifying a first subset ofthe blacklist term set in the medical report by searching the text ofthe medical report for the blacklist term set and/or by determining ifeach term in the text of the medical report is included in the blacklistterm set. Performing the second patient identifier detection functioncan further include identifying at least one term in the medical reportthat is included in the whitelist term set, and determining to ignorethe term in response. The first subset can be anonymized to generate thede-identified medical report as discussed herein. New patientidentifiers not already found can be appended to the blacklist term set,and the updated blacklist term set can be applied to perform a secondsearch of the header and/or image data of the medical scan, and at leastone of the new patient identifiers can be identified in the header inthe second search of the header and/or in the image data in a secondsearch of the image data. These newly identified patient identifiers inthe header and/or image data are anonymized in generating thede-identified medical scan.

As another example, a second subset of the blacklist term set can bedetected in a set of regions of image data of the medical scan byperforming the medical scan image analysis function on image data of themedical scan, where the image analysis function includes searching theimage data for the set of patient identifiers. For example, the medicalscan image analysis function can include searching the image data fortext, and the second subset can include detected text that matches oneor more terms of the blacklist term set. In some embodiments, detectedtext that matches one or more terms of the whitelist term set can beignored. The second subset can be anonymized to generate de-identifiedimage data as discussed herein. New patient identifiers that aredetected can be appended to the blacklist term set, and the updatedblacklist term set can be applied to perform a second search of theheader and/or metadata of the medical scan, and/or can be applied toperform a second search of the medical report. At least one of the newpatient identifiers can be identified in the header as a result ofperforming the second search of the header and/or at least one of thenew patient identifiers can be identified medical report as a result ofperforming the second search of the medical report. These newlyidentified patient identifiers can be anonymized in the header alongwith the originally identified blacklist term set in generating thede-identified header, and/or can be anonymized in the medical reportalong with the originally identified first subset in generating thede-identified medical report.

In some embodiments, the memory 2806 further stores a global blacklist,for example, that includes a vast set of known patient identifyingterms. In some embodiments, the global blacklist is also utilized by atleast one patient identifier detection function and/or in performing thesecond pass to determine patient identifying terms for anonymization. Insome embodiments, the blacklist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal blacklist for use in performing the second pass and/or indetecting patient identifiers in subsequently received medical scansand/or medical reports.

Alternatively or in addition, the memory 2806 can further store a globalwhitelist, for example, that includes a vast set of terms that can beignored. In particular, the global whitelist can include clinical termsand/or other terms that are deemed beneficial to preserve that do notcorrespond to patient identifying information. In some embodiments, theglobal whitelist is utilized by at least one patient identifierdetection function and/or in performing the second pass to determineterms to ignore in the header, image data, and/or medical report. Insome embodiments, the whitelist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal whitelist for use in performing the second pass and/or inignoring terms in subsequently received medical scans and/or medicalreports.

Alternatively or in addition, the memory 2806 can further store a globalgraylist, for example, that includes ambiguous terms that could bepatient identifying terms in some contexts, but non-identifying terms inother contexts. For example, “Parkinson” could correspond to patientidentifying data if part of a patient name such as “John Parkinson”, butcould correspond to non-patient identifying data meant to be ignored andpreserved in the de-identified medical report and/or de-identifiedmedical scan if part of a diagnosis term such as “Parkinson's disease.”In some embodiments, the global graylist is also utilized in performingthe second pass and/or in performing at least one patient identifierdetection function to determine that a term is included in the graylist,and to further determine whether the term should be added to theblacklist term set for anonymization or whitelist term set to be ignoredby leveraging context of accompanying text, by leveraging known datatypes of a header field from which the term was extracted, by leveragingknown structure of the term, by leveraging known data types of alocation of the image data from which the term was extracted, and/or byleveraging other contextual information. In some embodiments, thegraylist term set can be updated based on blacklist and/or whitelistterm sets for a particular medical scan and corresponding medicalreport.

In some embodiments, the at least one anonymization function includes afiducial replacement function. For example, some or all of the blacklistterm set can be replaced with a corresponding, global fiducial in theheader, report data, and/or image data. In some embodiments, the globalfiducial can be selected from a set of global fiducials based on a typeof the corresponding patient identifier. Each patient identifierdetected in the header and/or medical report can be replaced with acorresponding one of the set of global text fiducials. Each patientidentifiers detected in the image data can be replaced with acorresponding one of the set of global image fiducials. For example, oneor more global image fiducials can overlay pixels of regions of theimage data that include the identifying patient data, to obfuscate theidentifying patient data in the de-identified image data.

The global text fiducials and/or global image fiducials can berecognizable by inference functions and/or training functions, forexample, where the global text fiducials and global image fiducials areignored when processed in a training step to train an inference functionand/or are ignored in an inference step when processed by an inferencefunction. Furthermore, the global text fiducials and/or global imagefiducials can be recognizable by a human viewing the header, medicalreport, and/or image data. For example, a radiologist or other medicalprofessional, upon viewing a header, medical report, and/or image data,can clearly identify the location of a patient identifier that wasreplaced by the fiducial and/or can identify the type of patientidentifier that was replaced by the fiducial.

As an example, the name “John Smith” can be replaced in a header and/ormedical report with the text “% PATIENT NAME %”, where the text “%PATIENT NAME %” is a global fiducial for name types of the header and/orthe text of medical reports. The training step and/or inference step ofmedical scan natural language analysis functions can recognize andignore text that matches “% PATIENT NAME %” automatically.

FIG. 10B illustrates an example of anonymizing patient identifiers inimage data of a medical scan. In this example, the name “John Smith” andthe date “May 4, 2010” is detected as freehand text in the originalimage data of a medical scan. The regions of the image data that includethe patient identifiers can each be replaced by global fiducial in theshape of a rectangular bar, or any other shape. As shown in FIG. 10B, afirst region corresponding to the location of “John Smith” in theoriginal image data is replaced by fiducial 2820 in the de-identifiedimage data, and a second region corresponding to the location of “May 4,2010” in the original image data is replaced by fiducial 2822 in thede-identified image data. The size, shape, and/or location of eachglobal visual fiducial can be automatically determined based on thesize, shape, and/or location of the region that includes the patientidentifier to minimize the amount of the image data that is obfuscated,while still ensuring the entirety of the text is covered. While notdepicted in FIG. 10B, the fiducial can be of a particular color, forexample, where pixels of the particular color are automaticallyrecognized by the training step and/or inference step of medical scanimage analysis functions to indicate that the corresponding region beignored, and/or where the particular color is not included in theoriginal medical scan and/or is known to not be included in any medicalscans. The fiducial can include text recognizable to human inspectionsuch as “% PATIENT NAME” and “% DATE” as depicted in FIG. 10B, and/orcan include a QR code, logo, or other unique symbol recognizable tohuman inspection and/or automatically recognizable by the training stepand/or inference step of medical scan image analysis functions toindicate that the corresponding region be ignored.

In some embodiments, other anonymization functions can be performed ondifferent ones of the patient identifying subset of fields to generatethe de-identified header, de-identified report data, and/orde-identified image data. For example, based on the type of identifyingdata of each field of the header, different types of headeranonymization functions and/or text anonymization functions can beselected and utilized on the header fields, text of the report, and/ortext extracted from the image data. A set of anonymization functions caninclude a shift function, for example, utilized to offset a date, timeor other temporal data by a determined amount to preserve absolute timedifference and/or to preserve relative order over multiple medical scansand/or medical reports of a single patient. FIG. 10B depicts an examplewhere the shift function is performed on the date detected in the imagedata to generate fiducial 2822, where the determined amount is 10 yearsand 1 month. The determined amount can be determined by thede-identification system randomly and/or pseudo-randomly for eachpatient and/or for each medical scan and corresponding medical report,ensuring the original date cannot be recovered by utilizing a knownoffset. In various embodiments, other medical scans and/or medicalreports are fetched for the same patient by utilizing a patient IDnumber or other unique patient identifier of the header. These medialscans and reports can be anonymized as well, where the dates and/ortimes detected in these medical scans and/or medical reports offset bythe same determined amount, randomized or pseudo-randomized forparticular patient ID number, for example, based on performing a hashfunction on the patient ID number.

The set of anonymization functions can include at least one hashfunction, for example utilized to hash a unique patient ID such as apatient ID number, accession number, and/or SOP instance UID of theheader and/or text. In some embodiments, the hashed SOP instance UID,accession number, and/or patient ID number are prepended with a uniqueidentifier, stored in a database of the memory 2806 and/or shared withthe entities to which the de-identified medical scans and/or medicalreports are transmitted, so that de-identified medical scans and theircorresponding de-identified medical reports can be linked and retrievedretroactively. Similarly, longitudinal data can be preserved as multiplemedical scans and/or medical reports of the same patient will beassigned the same hashed patient ID.

The set of anonymization functions can further include at least onemanipulator function for some types of patient identifiers. Some valuesof header fields and/or report text that would normally not beconsidered private information can be considered identifying patientdata if they correspond to an outlier value or other rare value thatcould then be utilized to identify the corresponding patient from a verysmall subset of possible options. For example, a patient age over 89could be utilized to determine the identity of the patient, for example,if there are very few patients over the age of 89. To prevent suchcases, in response to determining that a patient identifier correspondsto an outlier value and/or in response to determining that a patientidentifier compares unfavorably to a normal-range threshold value, thepatient identifier can be capped at the normal-range threshold value orcan otherwise be manipulated. For example, a normal-range thresholdvalue corresponding to age can be set at 89, and generating ade-identified patient age can include capping patient ages that arehigher than 89 at 89 and/or can include keeping the same value forpatient ages that are less than or equal to 89.

In some embodiments, the de-identified header data is utilized toreplace the corresponding first subset of patient identifiers detectedin the medical report with text of the de-identified header fields. Inother embodiments, a set of text anonymization functions includes aglobal text fiducial replacement function, shift function, a hashfunction, and/or manipulator functions that anonymize the correspondingtypes of patient identifiers in the medical report separately.

In some embodiments where the image data of a medical scan includes ananatomical region corresponding to a patient's head, the image data mayinclude an identifying facial structure and/or facial features thatcould be utilized to determine the patient's identity. For example, adatabase of facial images, mapped to a corresponding plurality of peopleincluding the patient, could be searched and a facial recognitionfunction could be utilized to identify the patient in the database.Thus, facial structure included in the image data can be consideredpatient identifying data.

To prevent this problem and maintain patient privacy, thede-identification system can further be implemented to perform facialobfuscation for facial structure detected in medical scans. At least oneregion of the image data that includes identifying facial structure canbe determined by utilizing a medical image analysis function. Forexample, the medical image analysis function can include a facialdetection function that determines the regions of the image data thatinclude identifying facial structure based on searching the image datafor pixels with a density value that corresponds to facial skin, facialbone structure, or other density of an anatomical mass type thatcorresponds to identifying facial structure, and the facial obfuscationfunction can be performed on the identified pixels. Alternatively or inaddition, the facial detection function can determine the region basedon identifying at least one shape in the image data that corresponds toa facial structure.

The image obfuscation function can include a facial structureobfuscation function performed on the medical scan to generatede-identified image data that does not include identifying facialstructure. For example, the facial structure obfuscation function canmask, scramble, replace with a fiducial, or otherwise obfuscate thepixels of the region identified by the facial detection function. Insome embodiments, the facial structure obfuscation function can performa one-way function on the region that preserves abnormalities of thecorresponding portions of the image, such as nose fractures or facialskin legions, while still obfuscating the identifying facial structuresuch that the patient is not identifiable. For example, the pixels ofthe identifying facial structure can be altered such that they convergetowards a fixed, generic facial structure. In some embodiments, aplurality of facial structure image data of a plurality of patients canbe utilized to generate the generic facial structure, for example,corresponding to an average or other combination of the plurality offaces. For example, the pixels of the generic facial structure can beaveraged with, superimposed upon, or otherwise combined with the pixelsof the region of the image data identified by the facial detectionfunction in generating the de-identified image data.

In some embodiments, a hash function can be performed on an average ofthe generic facial structure and the identified facial structure of theimage data so that the generic facial structure cannot be utilized inconjunction with the resulting data of the de-identified image data toreproduce the original, identifying facial structure. In suchembodiments, the hash function can alter the pixel values while stillpreserving abnormalities. In some embodiments, a plurality of random,generic facial structures can be generated by utilizing the plurality offacial structure image data, for example, where each if the plurality offacial structure image data are assigned a random or pseudo-randomweight in an averaging function utilized to create the generic facialstructure, where a new, random or pseudo-random set of weights aregenerated each time the facial structure obfuscation function isutilized to create a new, generic facial structure to be averaged withthe identified facial structure in creating the de-identified image datato ensure the original identifying facial structure cannot be extractedfrom the resulting de-identified image data.

While facial obfuscation is described herein, similar techniques can beapplied in a similar fashion to other anatomical regions that aredetermined to include patient identifiers and/or to other anatomicalregions that can be utilized to extract patient identifying informationif not anonymized.

In some embodiments, the at least one receiver 2802 is included in atleast one transceiver, for example, enabling bidirectional communicationbetween the medical picture archive system 2620 and/or the reportdatabase 2625. In such embodiments, the de-identification system 2800can generate queries to the medical picture archive system 2620 and/orthe report database 2625 for particular medical scans and/or medicalreports, respectively. In particular, if the medical scan and medicalreport are stored and/or managed by separate memories and/or separateentities, they may not be received at the same time. However, a linkingidentifier, such as DICOM identifiers in headers or metadata of themedical scan and/or medical report, such accession number, patient IDnumber, SOP instance UID, or other linking identifier that maps themedical scan to the medical report can be utilized to fetch a medicalreport corresponding to a received medical scan and/or to fetch amedical scan corresponding to a received medical report via a query sentutilizing the at least one transceiver. For example, in response toreceiving the medical scan from the medical picture archive system 2620,the de-identification system can extract a linking identifier from aDICOM header of the medical scan, and can query the report database 2625for the corresponding medical report by indicating the linkingidentifier in the query. Conversely, in response to receiving themedical report from the report database 2625, the de-identificationsystem can extract the linking identifier from a header, metadata,and/or text body of the medical report, and can query the medicalpicture archive system 2620 for the corresponding medical scan byindicating the linking identifier in the query. In some embodiments, amapping of de-identified medical scans to original medical scans, and/ora mapping of de-identified medical reports to original medical reportscan be stored in memory 2806. In some embodiments, linking identifierssuch as patient ID numbers can be utilized to fetch additional medicalscans, additional medical reports, or other longitudinal datacorresponding to the same patient.

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system 2800 that stores executable instructions that,when executed by at least one processor, cause the de-identification toperform the steps below.

Step 2902 includes receiving from a first entity, via a receiver, afirst medical scan and a medical report corresponding to the medicalscan. Step 2904 includes identifying a set of patient identifiers in asubset of fields of a first header of the first medical scan. Step 2906includes performing a header anonymization function on each of the setof patient identifiers to generate a corresponding set of anonymizedfields. Step 2908 includes generating a first de-identified medical scanby replacing the subset of fields of the first header of the firstmedical scan with the corresponding set of anonymized fields. Step 2910includes identifying a first subset of patient identifiers of the set ofpatient identifiers in the medical report by searching text of themedical report for the set of patient identifiers. Step 2912 includesperforming a text anonymization function on the first subset of patientidentifiers to generate corresponding anonymized placeholder text foreach of the first subset of patient identifiers. Step 2914 includesgenerating a de-identified medical report by replacing each of the firstsubset of patient identifiers with the corresponding anonymizedplaceholder text. Step 2916 includes transmitting, via a transmitter,the de-identified first medical scan and the de-identified medicalreport to a second entity via a network.

In various embodiments, the medical scan is received from a PictureArchive and Communication System (PACS), where the medical report isreceived from a Radiology Information System (RIS), and where the firstde-identified medical scan and the de-identified medical report aretransmitted to a central server that is not affiliated with the PACS orthe RIS. In various embodiments, first medical scan and the medicalreport are stored in a first memory for processing. The first memory isdecoupled from the network to prevent the set of patient identifiersfrom being communicated via the network. The first de-identified medicalscan and the de-identified medical report are stored in a second memorythat is separate from the first memory. The first de-identified medicalscan and the de-identified medical report are fetched from the secondmemory for transmission to the second entity.

In various embodiments, the header anonymization function performed oneach of the set of patient identifiers is selected from a plurality ofheader anonymization functions based on one of a plurality of identifiertypes of the corresponding one of the subset of fields. In variousembodiments, the plurality of identifier types includes a date type. Ashift function corresponding to the date type is performed on a firstdate of the first header to generate the first de-identified medicalscan, where the shift function includes offsetting the first date by adetermined amount. A second medical scan is received, via the receiver,that includes a second header. A unique patient ID of the first headermatches a unique patient ID of the second header. The shift function isperformed on a second date of the second header by offsetting the seconddate by the determined amount to generate a second de-identified medicalscan. The second de-identified medical scan is transmitted to the secondentity via the network.

In various embodiments, the plurality of identifier types includes aunique patient ID type. A hash function corresponding the unique patientID type is performed on the unique patient ID of the first header togenerate the first de-identified medical scan. The hash function isperformed on the unique patient ID of the second header to generate thesecond de-identified medical scan. An anonymized unique patient ID fieldof the first de-identified medical scan matches an anonymized uniquepatient ID field of the second de-identified medical scan as a result ofthe unique patient ID of the first header matching the unique patient IDof the second header.

In various embodiments, the plurality of identifier types includes alinking identifier type that maps the medical scan to the medicalreport. A hash function corresponding to the linking identifier type isperformed on a linking identifier of the first header to generate ahashed linking identifier. A linking identifier field of the firstde-identified medical scan includes the hashed linking identifier.Performing the text anonymization function on the first subset ofpatient identifiers includes determining one of the first subset ofpatient identifiers corresponds to linking identifier text andperforming the hash function on the one of the first subset of patientidentifiers to generate the hashed linking identifier, where thede-identified medical report includes the hashed linking identifier.

In various embodiments, a second subset of patient identifiers of theset of patient identifiers is identified in a set of regions of imagedata of the medical scan by performing an image analysis function onimage data of the medical scan. The image analysis function includessearching the image data for the set of patient identifiers. Anidentifier type is determined for each of the second subset of patientidentifiers. One of a plurality of image fiducials is selected for eachof the second subset of patient identifiers based on the identifiertype. De-identified image data is generated, where a set of regions ofthe de-identified image data, corresponding to the set of regions of theimage data, includes the one of the plurality of image fiducials toobfuscate each of the second subset of patient identifiers. Generatingthe first de-identified medical scan further includes replacing theimage data of the medical scan with the de-identified image data.

In various embodiments, a new patient identifier is identified in themedical report by performing a natural language analysis function on themedical report, where new patient identifier is not included in the setof patient identifiers. The set of patient identifiers is updated toinclude the new patient identifier prior to searching the image data ofthe medical scan for the set of patient identifiers, and the secondsubset of patient identifiers includes the new patient identifier.

In various embodiments, the memory further stores a global identifierblacklist. The natural language analysis function includes searching themedical report for a plurality of terms included in the globalidentifier blacklist to identify the new patient identifier. In variousembodiments, the de-identification system determines that the globalidentifier blacklist does not include one of the set of patientidentifiers, and the global identifier blacklist is updated to includethe one of the set of patient identifiers.

In various embodiments, performing the image analysis function furtherincludes identifying a new patient identifier in the image data, wherenew patient identifier is not included in the set of patientidentifiers. Identifying text is extracted from a region of the imagedata corresponding to the new patient identifier. The new patientidentifier is identified in the medical report by searching text of themedical report for the identifying text. The text anonymization functionis performed on new patient identifier to generate anonymizedplaceholder text for the new patient identifier. Generating thede-identified medical report further includes replacing the identifyingtext with the anonymized placeholder text for the new patientidentifier.

In various embodiments, generating the de-identified image data furtherincludes detecting an identifying facial structure in the image data ofthe medical scan. Generating the de-identified image data includesperforming a facial structure obfuscation function on the image data,and where the de-identified image data does not include the identifyingfacial structure.

FIGS. 12A-12G illustrate an embodiments of a medical scan hierarchicallabeling system 3002. The medical scan hierarchical labeling system 3002can be utilized to generate structured labeling data for medical scansvia one or more client devices 120, based on user input to aninteractive interface displayed on a display device corresponding to theone or more client devices.

As shown in FIGS. 12A-12G, the medical scan hierarchical labeling system3002 can communicate bi-directionally, via network 150, with the medicalscan database 342, medical scan analysis function database 346, and/orother databases of the database storage system 140, with one or moreclient devices 120, and/or, while not shown in FIG. 12A, one or moresubsystems 101 of FIG. 1. In some embodiments, the medical scanhierarchical labeling system 3002 is an additional subsystem 101 of themedical scan processing system 100, implemented by utilizing thesubsystem memory device 245, subsystem processing device 235, and/orsubsystem network interface 265 of FIG. 2A. In some embodiments, themedical scan hierarchical labeling system 3002 is implemented byutilizing, or otherwise communicates with, the central server 2640. Forexample, some or all of the databases of the database storage system 140are populated with de-identified data generated by the medical picturearchive integration system 2600. In some embodiments, the medical scanhierarchical labeling system 3002 can receive de-identified medicalscans, annotation data, and/or reports directly from the medical picturearchive integration system 2600. For example, the medical scanhierarchical labeling system 3002 can request de-identified medicalscans, annotation data, and/or reports that match requested criteria,for example, corresponding to training set criteria. In someembodiments, some or all of the medical scan hierarchical labelingsystem 3002 is implemented by utilizing other subsystems 101 and/or isoperable to perform functions or other operations described inconjunction with one or more other subsystems 101. In some embodiments,the medical scan hierarchical labeling system 3002 is integrated withinand/or utilizes the medical scan assisted review system 102 and/or themedical scan annotator system 106.

As shown in FIG. 12A, the medical scan hierarchical labeling system 3002can store labeling application data 3020 of a labeling applicationassociated with the medical scan hierarchical labeling system 3002. Thelabeling application data 3020 can include a plurality of promptdecision trees. The plurality of prompt decision trees can include adiagnosis prompt decision tree 3022, a characterization prompt decisiontree 3024, and/or a localization prompt decision tree 3026. The labelingapplication data 3020 can be sent to the one or more client devices. Thelabeling application data can be stored as a client application asillustrated in FIG. 2A, stored in client memory device 240. The clientprocessing device 230 can execute operational instructions of thelabeling application data to enable the corresponding client device 120to run the labeling application. The labeling application can include aninteractive interface 3075 displayed on display device 270.

The client device can further receive a medical scan from the medicalscan for labeling, for example, as a transmission from the medical scanhierarchical labeling system 3002, fetched directly from the medicalscan database 342, and/or uploaded to the client device directly. Asshown in FIG. 12B, the client device can utilize the labelingapplication to generate labeling data for the medical scan fortransmission back to the medical scan hierarchical labeling system 3002.The labeling data can be mapped to the medical scan in the medical scandatabase, and can correspond to some or all fields of a medical scanentry 352, such as diagnosis data 440.

The medical scan database can correspond to a relational database and/ora database with a structured set of fields for the plurality of medicalscan entries. The labeling data generated via the labeling applicationcan correspond to the structured set of fields. In particular, each ofthe plurality of prompt decision trees can include leaf nodes thatcorrespond to the structured set of fields, and the labeling applicationcan display prompts to a user of the client device in accordance withthe plurality of prompt decision trees. The labeling application cangenerate labeling data that corresponds to leaf nodes of the pluralityof prompt decision tree, based on user input to the prompts indicatingselections from sets of selection options that correspond to subsets ofthe structured set of fields. Requiring that all labeling data adheresto a uniform structure with a discrete set of possibilities in thisfashion allows the labeling data to be consumable and easily utilized byother systems. For example, the labeling data can be utilized astraining data for one or more other subsystems 101 train a medical scananalysis function.

The labeling application can be utilized by users such as radiologistsand/or other labelers responsible for labeling and/or annotating medicalscans of the medical scan database with diagnosis data. The interactiveinterface 3075 can display image data of the medical scan to such a userin conjunction with a plurality of prompts to provide diagnosis,characterization, and/or localization labeling of the medical scan forat least one abnormality identified by the user. The plurality ofprompts can present a fixed set of differential diagnosis optionsallowing the user to select one or more of the differential diagnosisoptions corresponding to one or more abnormalities detected by the userin the medical scan. The plurality of prompts can present a fixed set ofcharacterization options to classify, describe, or otherwisecharacterize the one or more differential diagnoses identified in thefixed set of diagnosis data options. The plurality of prompts canpresent a fixed set of localization options to indicate a region ofinterest and/or specific anatomical location associated with the one ormore differential diagnoses identified in the fixed set of diagnosisdata options.

The fixed sets of diagnosis options, characterization option, and/orlocalization options can correspond to a fixed set of hierarchicaloptions, and can be characterized by a diagnosis prompt decision tree, acharacterization prompt decision tree, and/or a localization promptdecision tree, respectively. FIGS. 12C, 12D, and 12E illustrate examplesof a diagnosis prompt decision tree 3022, a characterization promptdecision tree 3024, and/or a localization prompt decision tree 3026,respectively. As illustrated, the diagnosis prompt decision tree 3022,the characterization prompt decision tree 3024, and/or the localizationprompt decision tree 3026 can each include a root node, a plurality ofinternal nodes that each branch from the root node or another internalnode, and a plurality of leaf nodes that each branch from the root nodeor an internal node. In particular, all of the fixed set of hierarchicaldiagnosis options, characterization options, and/or localization optionscan be represented by all of the leaf nodes of the diagnosis promptdecision tree, the characterization prompt decision tree, and/or thelocalization prompt decision tree, respectively. Each root node andinternal node can include any number of branches corresponding to aplurality of options that can be selected in response to a promptcorresponding to the node. Each node can be located in one of aplurality of levels of the corresponding prompt decision tree, whereeach level is characterized by a number of branches from the root node.

The plurality of prompt decision trees can be stored in the applicationdata as a data structures and/or abstract data type corresponding totrees in accordance with the structure of the corresponding decisiontrees. In some embodiments, other data structures and/or abstract datatypes are employed that indicate the plurality of decision trees and/orthat cause the labeling application to select a set of prompts that arepresented to the user as dictated by a corresponding prompt decisiontree, to select a corresponding set of options for each prompt asdictated by a corresponding prompt decision tree, and/or to select anordering of the set of prompts as dictated by a corresponding promptdecision tree, based on each user selection from the selected set ofoptions. This can include filtering a total plurality of prompts and/ora total plurality of options based on user selections, in accordancewith the corresponding decision tree. Thus, the labeling application canutilize a prompt selection algorithm that presents the prompts andoptions in accordance with at least one corresponding decision tree,even if the application data does not include any tree data structures.The examples presented in FIGS. 12C, 12D, and 12E present the behaviorexhibited in execution of the labeling application by illustrating theprompt selection algorithm for selecting for the set of prompts that arepresented to the user, the corresponding set of options for each prompt,and the ordering of the set of prompts. While the organization of datathat includes these prompts and/or options can be stored in acorresponding tree structure in the application labeling data, any otherdata structure can be employed to organize storage of these promptsand/or options. Furthermore, any other data structure can be employed toorganize storage of the instructions for such a prompt selectionalgorithm for selecting the prompts, the ordering of the prompts, and/oroptions associated with the prompts as discussed herein.

In presenting these prompt selection algorithms, FIGS. 12C-12Eillustrate the distinction between different internal nodes and leafnodes in accordance with the unique path from the root prompt. Forexample, diagnosis prompt 1.2 corresponds to the internal node reachedafter selecting selection option 1 from a set of selection options 1-Nfrom the diagnosis root prompt to reach characterization prompt 1, andthen selecting selection option 1.2 from a set of selection options1.1-1.M. As presented in FIGS. 12C-12E, M, N, and R correspond to anyinteger number of branches from the corresponding node. The use ofellipses “[ . . . ]” as presented in FIGS. 12C-12E indicates that anynumber of additional selections were made to reach the correspondingnode, and that the corresponding node can be at any level of the promptdecision tree. For example, leaf node 1.M.[ . . . ] indicates a leafnode that extends any number of branches from prompt 1.M. The promptdecision trees described herein can be of the same or differentconfigurations presented in FIGS. 12C-12E.

The labeling application can require that a leaf node is reached in eachprompt decision tree presented to the user, for example, where theinteractive interface will not advance to a next prompt decision treeand/or will not exit until the user continues to make selections toultimately advance to a leaf node. This ensures that the user'sannotation of the medical fully characterizes at least one abnormalityof the medical scan, while still producing structured, consumablelabeling data.

Some or all of nodes of one or more prompt decision trees can correspondto a prompt presented to a user via the interactive interface, where aselected branch from the node to a next node away from the root node isdetermined based on user input corresponding to selection of an optioncorresponding to the branch. Furthermore, some or all nodes of one ormore prompt decision trees can correspond to options presented to thelabeling application internally, where selected branches from thesenodes can be automatically determined for example, based on classifierdata 420 of the scan itself such as the modality of the scan and/oranatomical region of the scan; based on leaf nodes reached in of otherprompt decision trees; and/or based on another automatic determinationthat does not correspond to user input to the interactive interface.This can be utilized to advance from the root node to an internal nodeautomatically, where a user is presented with the plurality of promptsof a prompt decision tree starting from this automatically selectedinternal node. This can also be utilized to automatically advance to aninternal node or leaf node based on automatic determinationscorresponding to selections of branches to advance down the decisiontree, where the automatic determinations are made by the labelingapplication without user input.

The labeling application can include a plurality of diagnosis promptdecision trees, a plurality of characterization prompt decision trees,and a plurality of localization prompt decision trees for each modality,each anatomical regions, and/or for other scan classifier data 420. Forexample head CTs, chest x-rays, and chest CTs can each correspond totheir own diagnosis prompt decision tree, a characterization promptdecision tree, and localization prompt decision tree. In suchembodiments, the labeling application can automatically determine amodality, anatomical region, and/or other category of the medical scan,and can further automatically determine which of the plurality of treeswill be used based on the determined category.

The interactive interface can display each of a plurality of prompts toa user of the client device, one at a time, in accordance with thediagnosis prompt decision tree, characterization prompt decision tree,and/or localization prompt decision tree of the labeling applicationdata 3020. The user can select one of a fixed set of options indicatedby each of the plurality of prompts of the diagnosis prompt decisiontree as user input via interaction with the interactive interface 3075.The selected one of the plurality of options can dictate the next one ofthe plurality of prompts, in accordance with the corresponding promptdecision tree. These prompts can continue to be displayed in sequence tothe user, progressing to the next prompt indicated by the promptdecision tree as the user selects one of the plurality of optionspresented in accordance with each prompt, until a leaf node of theprompt decision tree is reached. The leaf node can indicate some or allof the selections made by the user in previous nodes of the promptdecision tree from a root node to the leaf node.

Furthermore, each set of options presented for a prompt corresponding toan internal node can include only an appropriate set of options that arepossible given the previous selections corresponding to branches fromthe root node to the diagnosis prompt node. This effectively filters theset of options presented to the user given the selections made so far,ensuring the set of presented options includes only valid options. Theseappropriate sets of options can be predetermined by an administrator orother user responsible for creating the prompt decision trees. In someembodiments, the medical scan hierarchical labeling system 3002automatically determines the options presented for each prompt of adecision tree based on a known set of rules, for example, correspondingto possible options given modality, anatomical regions, and userselections that must be made to reach the corresponding node. In someembodiments, the medical scan hierarchical labeling system 3002automatically optimizes an ordering of some or all of the prompts. Insome embodiments, identical prompts are included in multiple paths of aprompt decision tree, but are presented in different orders. In someembodiments, an ordering of prompts to each leaf node is determined tooptimize an expected number of branches necessary to reach a leaf node,optimize the average number of branches to reach a leaf node, to reduceor otherwise optimize the number of selection options presented in oneor more prompts of one or more nodes, and/or to otherwise optimize theprompt decision tree. In some embodiments, the set of prompts to reacheach leaf node is automatically determined to ensure the correspondingabnormality of each leaf node will be fully described in reaching theleaf node. In some embodiments, the number of this set of prompts isminimized, while still ensuring the corresponding abnormality of eachleaf node will be described to a necessary extent in reaching the leafnode.

The interactive interface can first present the user with a promptasking whether or not an abnormality is present, as shown in FIG. 12C.If the user selects “no”, the diagnosis prompt decision tree canimmediately branch to a leaf node indicating no abnormality is present,and the labeling data can correspond to a structured entry of themedical scan database indicating no abnormality is present. If the userselects the option indicating an abnormality is present, the interactiveinterface can display a diagnosis root prompt with a plurality ofselection options 1-N. The user can select a single one of the Nselection options to progress to the corresponding next diagnosis prompt1-N. For example, the user can select option 1, and the interactiveinterface will display diagnosis prompt 1 in response, indicating a setof selection options 1.1-1.M. If the user selects option 1.1.1 inresponse to diagnosis prompt 1.1, leaf node 1.1.1 is reached. Labelingdata corresponding to the diagnosis can indicate the diagnosiscorresponding to leaf node 1.1.1. The leaf node can effectively indicateall of the selections made from the root node to reach the leaf node.Thus, the labeling data generated in response to selection of the leafnode can further indicate each of the individual selections made by theuser to reach the leaf node, for example, where the labeling dataindicates selection option 1, selection option 1.1, and selection option1.1.1.

Each selection option 1, 1.1, and 1.1.1 can correspond to its own fieldof the relational database, and/or the relational database can includefields corresponding to only the set of leaf nodes, for example, with abinary indication of whether or not the diagnosis corresponding to eachleaf node is present. In some embodiments, the database includes asingle diagnosis field populated by one of the fixed set of optionsindicated by the one of the plurality of leaf nodes of the diagnosisprompt tree. The medical scan database can employ any structure with adiscrete set of possible diagnosis entries corresponding to acorresponding discrete set of diagnosis options corresponding to thediscrete set of leaf nodes of the diagnosis decision tree.

In some embodiments, the labeling application can automaticallydetermine a starting node of the diagnosis decision tree, correspondingto one of the interior nodes rather than the root node. In particular,the labeling application can automatically determine the starting nodeof the diagnosis decision tree based a determined modality of themedical scan, based on a determined anatomical region of the medicalscan, and/or other scan classifier data 420 of the medical scan. Leafnodes branching from the selected starting node will only includediagnosis options corresponding to abnormalities that can be detected inthe corresponding anatomical region of the medical scan and/or that canbe detected in the corresponding modality of the medical scan.

For example, the labeling application automatically proceed to selectionoption 1 that indicates plurality of abnormalities associated with thehead in response to determining that the medical scan is a head CT scan.In such embodiments, diagnosis prompt 1 will only present optionscorresponding to abnormalities that correspond to the head, and that canbe detected in a head CT. Any leaf nodes 1.[ . . . ] that branch fromdiagnosis prompt 1 will only correspond to final diagnosis thatcorrespond to the head, and that can be detected in a head CT. Forexample, the fixed set of options indicated by leaf nodes 1.[ . . . ]can include “brain tumor”, and “brain bleed”, but will not include“wrist fracture” or “pulmonary embolism.”

As another example, the labeling application can proceed automaticallyto a selection option 2.1 in response to determining that the medicalscan is a chest x-ray. For example, a selection option from thediagnosis root prompt can correspond to selecting the anatomical regionof the medical scan from a set of anatomical regions, where selectionoption 2 corresponds to the chest. Diagnosis prompt 2 can correspond toselecting the modality from a plurality of modalities, where selectionoption 2.1 corresponds to an x-ray. In such embodiments, diagnosisprompt 2.1 will only present options corresponding to abnormalities thatcorrespond to the chest, and that can be detected in an x-ray of thechest. Any leaf nodes 2.1.[ . . . ] that branch from diagnosis prompt2.1 will only correspond to final diagnosis that correspond to thechest, and can be detected in a chest x-ray. For example, the fixed setof options indicated by leaf nodes 2.1. [ . . . ] can include “ribfracture” or “pneumonia” because they can be identified by a chestx-ray. However, “pulmonary embolism” is not included in the leaf nodes2.1.[ . . . ], even though this condition is associated with the chest,because leaf nodes 2.1. [ . . . ] correspond to an x-ray modality, andbecause reviewing an x-ray alone is not sufficient to determine whethera pulmonary embolism is present or absent. However, if selection option2.2 corresponds to a CT scan, any leaf nodes 2.2.[ . . . ] that branchfrom diagnosis prompt 2.2 will only correspond to a final diagnosis thatcorresponds to the chest and that can be detected in a chest CT. Thus,leaf nodes 2.2.[ . . . ] can include “pulmonary embolism” as a selectionoption because this condition can be determined to be present or absentin reviewing a chest CT.

In some embodiments, a medical scan entry can include any variablenumber of differential diagnosis, where each of the variable number ofdifferential diagnosis are included in the same field or the variablenumber of different fields, and where each of the variable number ofdifferential diagnosis corresponds to a different one of the discreteset of leaf nodes of the diagnosis decision tree. In such embodiments,after reaching a leaf node of the diagnosis decision tree, the user canbe prompted with a question of whether or not there is anotherabnormality to characterize. In response to the user selecting “yes,”the interactive interface can return to the root prompt, and/or thestarting node determined based on the anatomical region and/or modality.Furthermore, a deeper starting node, branching from the initial startingnode but further away from the root node, can be selected automaticallyas a result of selection of the first one or more leaf nodes that havealready been selected, if these first one or more leaf nodes can narrowthe set of options possible for additional differential diagnoses. Insome embodiments, characterization and/or localization of theabnormality indicated in the first leaf node is completed in accordancewith the characterization prompt decision tree and/or the localizationprompt decision tree first, and once this characterization and/orlocalization is completed, the user is presented the option to identify,characterize, and/or localize additional abnormalities as differentialdiagnoses.

Once a leaf node is reached in the diagnosis decision tree and/or onceit is indicated that no further abnormalities are present, theinteractive interface can advance to prompts of the characterizationprompt decision tree. FIG. 12D illustrates an example embodiment of acharacterization prompt decision tree. The interactive interface canbegin with the root node of the characterization prompt decision treeand/or the labeling application can automatically advance to an internalnode based on the anatomical region and/or modality as discussed inconjunction with the diagnosis prompt decision tree Furthermore, as theset of options that can be used to characterize an abnormality dependson the type of abnormality itself, the labeling application canautomatically advance to an even deeper internal node based on the leafnode of the diagnosis decision tree that was ultimately selected. Forexample, as shown in FIG. 12C, a characterization prompt X progressingfrom leaf node 1.1.1 can correspond to any characterization prompt nodeof the characterization prompt decision tree determined to be anautomatic starting node given a diagnosis corresponding to leaf node1.1.1. Other leaf nodes of the diagnosis prompt decision tree can alsocause the interactive interface to automatically progress to their ownpre-determined starting node of the characterization prompt decisiontree.

From the selected starting node of the characterization prompt decisiontree, the corresponding diagnosis can be characterized based onselections from the starting node, where the interactive interfacepresents successive prompts corresponding to the nodes branching basedon selections to previous prompts, until a leaf node of thecharacterization prompt decision tree is reached. In embodiments wheremultiple leaf nodes for multiple abnormalities have already beenselected from the diagnosis prompt decision tree, each of theseabnormalities can be characterized separately, each with their ownautomatically determined starting node based on the corresponding leafnode, where a separate leaf node of the characterization decision treeis ultimately reached for each of these differential diagnoses.

The user can indicate if one or more differential diagnoses, identifiedvia selection of multiple leaf nodes of the diagnosis prompt decisiontree, correspond to side effects of a main one of the differentialdiagnoses. This can be performed in the plurality of characterizationprompts, where one or more side effects are indicated and characterizedas part of characterizing the main diagnosis. As one example,illustrated in FIG. 12D, once a diagnosis is characterized and a leafnode is reached, the user will be prompted with a question askingwhether or not the diagnosis is further characterized by any sideeffects. If so, a selected starting diagnosis prompt Y of the diagnosisprompt decision tree will be presented, and the interactive interfacewill present prompts of the diagnosis prompt decision tree until a leafnode of the diagnosis prompt decision tree is ultimately selected foreach side effect. Furthermore, each side effect can be further describedvia selection of leaf nodes in the characterization prompt decision treeand/or the localization prompt decision tree for the each side effect.As another example, after the user identifies a plurality ofdifferential diagnoses, the interactive interface can present a promptto select at least one main diagnosis from the plurality of differentialdiagnoses, and can further present a prompt to identify at least one ofthe remaining differential diagnoses as side effects of the maindiagnosis.

For example, if a head CT includes a brain tumor, brain bleed, andfracture, the brain bleed and fracture can be indicated as side effectsof the brain tumor. The user can reach leaf nodes of the diagnosisdecision tree corresponding to each of “brain tumor”, “brain bleed”, and“fracture”. In particular, after the “brain tumor” leaf node is reachedin the diagnosis prompt decision tree, the user can be prompted toidentify one or more side effects of the brain tumor as part ofcharacterizing the brain tumor via the characterization prompt decisiontree. In response to the user selecting to identify a side effect,prompts of the diagnosis prompt decision tree will be displayed untilthe leaf node corresponding to “brain bleed” is ultimately reached. Inresponse to the user selecting to identify an additional side effect,prompts of the diagnosis prompt decision tree will be displayed untilthe leaf node corresponding to “fracture” is ultimately reached. Inresponse to the user selecting that no more side effects are present,the user can continue characterizing and/or advance to localizing thebrain tumor itself. The brain bleed and fracture can also each becharacterized and localized by ultimately reaching leaf nodes of thecharacterization prompt decision tree and localization prompt decisiontree for each of these side effects.

Once a leaf node is reached in the characterization decision tree and/oronce the user indicates that no further side effects are present, theinteractive interface can advance to prompts of the localization promptdecision tree 3026, presented in FIG. 12E. The interactive interface canbegin with the root node of the localization prompt decision tree and/orthe labeling application can automatically advance to an internal nodebased on the anatomical region and/or modality as discussed inconjunction with the diagnosis prompt decision tree and thecharacterization prompt decision tree. Furthermore, as the set ofoptions that can be used to localize an abnormality can depend on thetype of abnormality itself, the labeling application can automaticallyadvance to an even deeper internal node based on the leaf node of thediagnosis prompt decision tree that was ultimately selected, and/orbased on the leaf node of the characterization prompt decision tree thatwas ultimately selected. For example, as shown in FIG. 12D, alocalization prompt Z progressing from leaf node 1.2.R can correspond toany localization prompt node of the localization prompt decision treedetermined to be an automatic starting node given a diagnosis and/orcharacterization corresponding to leaf node 1.2.R. Other leaf nodes ofthe characterization prompt decision tree can also cause the interactiveinterface to automatically progress to their own pre-determined startingnode of the localization prompt decision tree.

From the selected starting node of the localization prompt decisiontree, the corresponding abnormality can be localized based on selectionsfrom the starting node, where the interactive interface presentssuccessive prompts corresponding to the nodes branching based onselections to previous prompts, until a leaf node of the localizationprompt decision tree is reached. In embodiments where multiple leafnodes for multiple abnormalities have already been selected from thediagnosis prompt decision tree as differential diagnoses and/or sideeffects, each of these abnormalities can be localized separately, eachwith their own automatically determined starting node based on thecorresponding leaf node, where a separate leaf node of the localizationdecision tree is ultimately reached for each of these abnormalities.

In some embodiments, as an abnormality might be located in multipleplaces, the user can select any number of leaf nodes to localize eachabnormality. For example, as shown in FIG. 12E, after reaching a leafnode such as leaf node 1.1.1, the user can be presented with an optionto identify other locations in which the abnormality is present. Inresponse to the user selecting “yes,” the interactive interface canreturn to the root prompt, and/or the starting node determined based onthe anatomical region and/or modality. Furthermore, a deeper startingnode, branching from the initial starting node but further away from theroot node, can be selected automatically as a result of selection of thefirst one or more leaf nodes of the localization prompt decision treethat have already been selected, if these first one or more leaf nodescan narrow the set of options possible for additional locationsdiagnoses, such as narrowing to a set of adjacent locations or locationswithin proximity of locations selected thus far.

At least one internal node of the localization prompt decision tree canrequire that multiple leaf nodes are reached to characterize theabnormality. For example, as shown in FIG. 12G, localization prompt 1indicates that a number Q leaf nodes are required. Once a leaf node,such as leaf node 1.2.1 is reached, if the required number Q leaf nodeshave not yet been selected, localization prompt 1 will be presented andthe user can progress to leaf nodes, returning to localization prompt 1until the required number of leaf node selections Q have been made.While the diagnosis prompt decision tree of FIG. 12C nor thecharacterization prompt decision tree of FIG. 12D illustrate internalnodes or root nodes requiring multiple leaf nodes be reached, such nodesrequiring multiple leaf node selections can similarly be included in thediagnosis prompt decision tree or the characterization prompt decisiontree in other embodiments.

In some embodiments, this required number of leaf nodes can inherentlybe built into the localization prompt decision tree with single requiredselection from each node. For example, instead of returning tolocalization prompt 1 from leaf node 1.2.1, the leaf node 1.2.1 caninstead be an internal node 1.2.1 presenting localization prompt 1, withthe same set of M selection options this internal node, allowing theuser to continue to eventually reach the Q leaf node selections as aresult of progressing down deeper sets of branches and identifying eachof the Q leaf nodes as selections to the final, Qth leaf node along theway.

As a particular example, if a leaf node corresponding to “brain tumor”is selected from the diagnosis prompt decision tree, at least one lobeand at least one compartment must be selected in the step of localizingthe brain tumor. The localization prompt decision tree can present theuser with a plurality of lobe selection options and a plurality ofcompartment selection options, and can require that at least one leafnode corresponding to a lobe is selected, and that at least one leafnode corresponding to a compartment is selected. For example, once alobe is selected, the same prompt can be presented, excluding the lobethat has already been selected, until one of the plurality ofcompartment options is also selected. As another example, thelocalization prompt decision tree can present a first prompt with only aplurality of lobe options in an internal node, and once one of theplurality of lobe options is selected, can branch to the next node thatpresents a second prompt with only a plurality of compartment options,where the leaf node ultimately reached indicates the selected lobe andthe selected compartment. In such embodiments, each of the plurality oflobe selection options can branch to the identical prompt nodespresenting the plurality of compartment options.

In some embodiments, based on the diagnosis leaf node, characterizationleaf node, and/or localization leaf node, an additional set of questionsmay be necessary. For example, when leaf node N. [ . . . ] is reached,at least one additional prompt will be presented. This additional set ofquestions can correspond to prompts of at least one additional promptdecision tree, where the questions are hierarchical and dependent onprevious questions, and/or where leaf nodes of each additional promptdecision tree must be reached. The additional prompt set can be the sameor different for different leaf nodes. The additional prompt set candepend only on the diagnosis leaf node, characterization leaf node,localization leaf node, and/or on a combination of two or more nodes.The additional prompt set can be further based on differential diagnosesor side effects. The additional prompt set can be based on the modalityof the medical scan and/or the anatomical region of the medical scan.For example, if the medical scan corresponds to a head CT, theadditional question set includes ventricular system questions that willautomatically be presented via the interactive interface in response toan automatic determination that the medical scan is a head CT. In someembodiments, the additional prompt set is inherently included asadditional internal nodes of the diagnosis prompt decision tree,characterization prompt decision tree, and/or localization promptdecision tree, ultimately reaching a final leaf node that includesanswers to the additional questions. The answers to the additionalquestions can be included in the fixed format of the labeling data,where each of the additional questions similarly presents a fixed set ofoptions. Alternatively or in addition, at least one additional questioncan correspond to unstructured data, such as text or voice input by theuser, drawings and/or or shapes outlining one or more abnormalitiessuperimposed upon the medical scan as user input entered by the user,measurement data indicating size, shape, diameter, and/or volume ofabnormalities as identified by the user, a report entered by the user,or other unstructured data. In some embodiments, this unstructured datacan be mapped to the medical scan in the database, but can be separatefrom the rest structured labeling data.

In some embodiments, the medical scan includes a plurality of imageslices. The user can be prompted to select a proper subset of slicesand/or a single slice that includes an abnormality to be described inthe labeling data. A user selection that indicates a selected subset ofthe plurality of images slices of the medical scan can be received viauser input to the interactive interface. This prompt can be included inthe localization prompt decision tree and can dictate further prompts ofthe localization prompt decision tree based on a narrowed anatomicallocation corresponding to the proper subset of slices and/or the singleslice. Alternatively, this prompt can be presented separately fromprompts of the localization prompt decision tree, and the startinglocalization prompt can be selected based on the selected subset of theplurality of image slices. For example, if a subset of slices selectedby the user indicates the frontal lobe of a head CT, localizationprompts presented by the user interface will include optionscorresponding to the selection of the frontal lobe.

The user can be prompted to provide an urgency ranking as part of aprompt decision tree, additional question set, and/or as a final promptpresented to the user. The urgency ranking prompt can similarly includea fixed set of urgency ranking options for selection by the user toindicate an urgency associated with the diagnosis, associated withfurther review of the medical scan, and/or associated with furtherscans, tests, or appointments with the patients that may be necessary.The urgency ranking can be included in the labeling data, and/or can beutilized in triaging of the medical scan by one or more subsystems.

In some embodiments, one or more of the nodes of one or more of theprompt decision trees can be optional, where a user selection is notrequired. Such nodes can include a “skip” branch, indicating thatselection of one or more of the selection options is not necessary forthis prompt. Selecting the skip option can correspond to a branch thatadvances to a next node in the prompt decision tree.

While the discussion thus far indicates that the user first selects aleaf node of the diagnosis prompt decision tree, then a leaf node of thecharacterization prompt decision tree, and finally a leaf node of thelocalization prompt decision tree, prompts of the diagnosis promptdecision tree, the characterization prompt decision tree, and thelocalization prompt decision tree can be presented in any order. Forexample, the user might first localize the abnormality, and then basedon this localization, a starting node of the diagnosis prompt decisiontree is determined based on types of abnormalities that can exist and/orbe observed at the particular location indicated in the localizationleaf node.

Furthermore, while FIGS. 12C-12E present the diagnosis prompt decisiontree, characterization prompt decision tree, and localization promptdecision tree separately, prompts corresponding to diagnosis,characterization, and localization of one or more abnormalities can beincluded in a single prompt decision tree of the labeling applicationdata, where reaching a leaf node of the single prompt decision treeincludes selecting at least one diagnosis option from at least oneinternal node corresponding to a diagnosis prompt, selecting at leastone characterization option from at least one internal nodecorresponding to a characterization prompt, and selecting at least onelocalization option from at least one internal node corresponding to alocalization prompt. In some embodiments, the flow of prompts to atleast one leaf node includes a plurality of diagnosis prompts,characterization prompts, and localization prompts in any order, forexample, where first a localization prompt is presented, then adiagnosis prompt, then another localization prompt, and then acharacterization prompt. In some embodiments, multiple prompt decisiontrees that includes this mix of diagnosis prompts, characterizationprompts, and/or localization prompts are included in the labelingapplication data. Each of these multiple prompt decision trees cancorrespond to a different modality, a different anatomical region, adifferent modality/anatomical region pair, and/or other different scanclassifier data 420.

The labeling application can be utilized by multiple client devicescorresponding to multiple users, and each user can label multiplemedical scans by utilizing the labeling application. Labeling datagenerated over time by one or more users for one or more medical scanscan tracked in a user database 344, where the number of scans labeled byeach user of each client device is tracked. The user database canfurther track how many scans have been labeled for each of a set of scancategories corresponding to different modalities, different anatomicalregions, and/or other categories indicated by scan classifiers data 420.Users can be incentivized and/or rewarded for reaching a thresholdnumber of labeled scans in one or more scan categories and/or reaching atotal number of labeled scans. Similarly, users can be can beincentivized and/or rewarded for reaching and/or maintaining a thresholdlabeling rate in one or more scan categories and/or reaching and/ormaintaining a threshold labeling rate overall. In some embodiments,users can be incentivized to label scans in categories with a countand/or labeling rate that is below a threshold for that user,encouraging the user to expand their skills and label scans ofmodalities and/or anatomical regions they do not typically label. Insome embodiments, users can be incentivized to label scans in categorieswith a count and/or labeling rate that is below a global thresholdacross all users, for example, corresponding to scan types that are notaddressed enough by users across the system, to encourage the user tomeet this need. The incentives can include financial incentives, caninclude a favorable adjustment to performance score data and/orqualification data, and/or can include advancing a user to an expertstatus in one or more scan categories in which a number and/or labelingrate compares favorably.

In some embodiments, a medical scan can be automatically pre-processedto partition the medical scan in accordance with multiple anatomicalregions included within the medical scan. For example, a full body scancan be partitioned into a set of medical scan portions, where eachmedical scan portion corresponds to each of a set of anatomical regions.For example, the full body scan can be partitioned into medical scanportions corresponding to the head, chest, arm, leg, etc. for individuallabeling. These partitions can each be labeled by the same user or bydifferent users. For example, the medical scan hierarchical labelingsystem 3002 can perform this pre-processing step prior to transmissionof the medical scan to a client device. The different medical scanportions can be sent to different users for labeling based ondetermining each user has favorable qualification data and/orperformance score data for the corresponding anatomical region. Thelabeling data can be retrieved from all of the users and can be compiledfor the original medical scan to be mapped to the medical scan database.In some embodiments, the pre-processing step is performed after amedical scan is retrieved by a client device as part of execution of thelabeling application. Each partition can be presented in conjunctionwith prompt decision trees corresponding to the anatomical region ofeach partition and/or in conjunction with starting nodes of the promptdecision trees determined based on the anatomical region of eachpartition.

As presented in FIGS. 12A and 12B, the labeling application is executedby the client device, allowing the client device to generate all of thelabeling data locally, and this final labeling data is then transmittedback to the medical scan hierarchical labeling system 3002 via thenetwork. In some embodiments, some or all of the steps performed by theclient device in accordance with execution with the labeling applicationcan be instead executed by the medical scan hierarchical labeling system3002. This can be accomplished via additional transmissions between themedical scan hierarchical labeling system 3002 and the client device.

For example, some or all of the user input, such as user selection of aselection option of a prompt, can be transmitted via the network themedical scan hierarchical labeling system 3002. The medical scanhierarchical labeling system 3002 can utilize the corresponding promptdecision tree, stored in memory of the medical scan hierarchicallabeling system 3002, to determine the next prompt that will bepresented to the user and the corresponding set of options. This nextprompt and corresponding set of options can be transmitted to the clientdevice for display via the interactive interface, where one of this setof options is selected by the user via user input, and is transmittedback to the medical scan hierarchical labeling system 3002. Another nextprompt and another next set of options is determined by medical scanhierarchical labeling system 3002 based on the prompt decision tree fortransmission back to the client device. This process can continue untila leaf node is ultimately selected.

As another example, the medical scan hierarchical labeling system 3002can automatically select the starting node for one or more of the promptdecision trees based on the anatomical region, modality, and/or otherfeatures of the medical scan, in conjunction with transmission of themedical scan. An indicator of the starting node can be transmitted tothe client device, and the client device can present the plurality ofprompts beginning with the starting node based on the indicator of thestarting node received from the medical scan hierarchical labelingsystem 3002.

As another example, the medical scan hierarchical labeling system 3002can automatically select the starting node for one or more of the promptdecision trees based on the anatomical region, modality, and/or otherfeatures of the medical scan that is transmitted to the client device. Asubset of the corresponding one or more prompt decision trees isselected by the medical scan hierarchical labeling system 3002, wherethe root node of the subset is the selected starting node, and where thesubset includes internal nodes and leaf nodes that extend from thestarting node. This subset of the one or more prompt decision trees cantransmitted to the client device for use, where the remainder of the oneor more prompt decision trees is not transmitted to the client deviceand/or not stored by the client device. This subset of the one or moreprompt decision trees can be utilized by the client device to presentthe plurality of prompts.

As shown in FIG. 12F, a medical scan training set can be retrieved fromthe medical scan database 342. The medical scan hierarchical labelingsystem 3002, or another subsystem 101, can perform a training step 3090,for example, by utilizing the medical scan image analysis system andperforming training step 1352 of FIG. 7A. The medical scan training setcan include medical scans that were labeled by one or more clientdevices by utilizing the medical scan hierarchical labeling system 3002as discussed in conjunction with FIGS. 12A-12E. In particular, labelingdata mapped to the medical scans of the medical scan training set wasgenerated by client devices based leaf nodes reached in the diagnosisprompt decision tree, characterization prompt decision tree, and/orlocalization prompt decision tree. This labeling data is indicated foreach medial scan in the medical scan training set, for example, as astructured entry in the medical scan database. The labeling data foreach medical scan in the training set can be utilized as an outputfeature vector and/or output nodes of a neural network in the trainingstep 3090, where the output feature vector and/or output nodes arestructured in accordance with the fixed structure of the labeling data.The input feature vector and/or input nodes of the neural network cancorrespond to image data of each medical scan in the training set,patient history data of each medical scan in the training set, and/orother data of the corresponding medical scan entry of each medical scanin the training set. Once the model is trained, model data can betransmitted to the medical scan function analysis database 346 to beutilized by one or more other subsystems 101. Alternatively or inaddition, the model data can be transmitted to the medical picturearchive integration system 2600 to be utilized in performance ofinference functions by the medical picture archive integration system2600.

The model data can be utilized by the medical scan hierarchical labelingsystem 3002, or another subsystem 101, as shown in FIG. 12G to generateinference labeling data for one or more new medical scans to be labeled,for example, retrieved from the medical scan database 342, by performingan inference function 3095 that utilizes the trained model. For example,the medical scan image analysis system 112 and/or the inference step1354, detection step 1372, and/or abnormality classification step 1374of FIG. 7B can be utilized to generate inference labeling data for newmedical scans. The inference labeling data can correspond to the samefixed structure of the labeling data generated by utilizing the labelingapplication, dictated by the fixed set of abnormality options, the fixedset of classification options, and the fixed set of localizationoptions. The inference labeling data can indicate probability values forone or more of the fixed set of abnormality options, one or more of thefixed set of classification options, and/or one or more of the fixed setof localization options. These probability values can indicateprobabilities that one or more of the fixed set of abnormality optionsis present in the scan and/or in a region of the scan corresponding to aprobability matrix corresponding to the probability value, that one ormore of the fixed set of classification options describes the one ormore of the fixed set of abnormality options, and/or that theabnormality is located in the one or more of the fixed set oflocalization options. The inference labeling data can be mapped to thenew medical scan in the medical scan database and/or can be transmittedto a client device for display to a user via a user interface.

In various embodiments, medical scan hierarchical labeling systemincludes a medical scan database that stores a plurality of medical scanentries, at least one processor, and a memory. The memory storeslabeling application data that includes application operationalinstructions and a plurality of prompt decision trees. The plurality ofprompt decision trees includes a diagnosis prompt decision tree, acharacterization prompt decision tree, and a localization promptdecision tree. Each of the plurality of prompt decision trees includes aroot node, a set of internal nodes, and a set of leaf nodes. Each rootnode and each of the set of internal nodes correspond to one of aplurality of prompts. Each root node and each of the set of internalnodes include a set of branches that each correspond to one of adiscrete set of selection options for the one of the plurality ofprompts. The memory further stores a medical scan relational databasethat stores a plurality of medical scan entries. The medical scanrelational database includes a discrete set of fields corresponding tothe leaf nodes of the plurality of prompt decision trees.

The executable instructions, when executed by the at least oneprocessor, cause the medical scan hierarchical labeling system totransmit, via a network, labeling application data to a client devicefor storage. The application operational instructions of the labelingapplication data, when executed by at least one client device processorof the client device, cause the client device to execute a labelingapplication. The executable instructions further cause the medical scanhierarchical labeling system to transmit via the network, a medical scanto the client device.

Execution of the labeling application by the client device causes theclient device to, in response to receiving the medical scan, display,via an interactive interface presented on a display device associatedwith the client device for display to a user associated with the clientdevice, image data of the medical scan. The client device automaticallydetermines a starting diagnosis prompt by selecting one of the set ofinternal nodes of the diagnosis prompt decision tree based on ananatomical region of the medical scan and further based on a modality ofthe medical scan. The client device displays, via the interactiveinterface, a plurality of diagnosis prompts of the diagnosis promptdecision tree, in succession, beginning with the starting diagnosisprompt, in accordance with corresponding nodes of the diagnosis promptdecision tree until a first one of the set of leaf nodes of thediagnosis prompt decision tree is ultimately selected. The interactiveinterface progresses to each next one of the plurality of diagnosisprompts by selecting one of the set of branches of each correspondingnode in accordance with each of a plurality of corresponding userdiagnosis selections, received via user input. Each of the plurality ofcorresponding user diagnosis selections corresponds to one of thediscrete set of selection options for each one of the plurality ofdiagnosis prompts displayed via the interactive interface.

The client device automatically determines a starting characterizationprompt by selecting one of the set of internal nodes of thecharacterization prompt decision tree based on the anatomical region ofthe medical scan, based on the modality of the medical scan, and furtherbased on the first one of the set of leaf nodes of the diagnosis promptdecision tree. The client device displays, via the interactiveinterface, a plurality of characterization prompts, in succession,beginning with the starting characterization prompt, in accordance withcorresponding nodes of the characterization prompt decision tree until afirst one of the set of leaf nodes of the characterization promptdecision tree is ultimately selected. The interactive interfaceprogresses to each next one of the plurality of characterization promptsby selecting one of the set of branches of each corresponding node inaccordance with each of a plurality of corresponding usercharacterization selections, received via user input. Each of theplurality of corresponding user characterization selections correspondsto one of the discrete set of selection options for each one of theplurality of characterization prompts displayed via the interactiveinterface.

The client device automatically determines a starting localizationprompt by selecting one of the set of internal nodes of the localizationprompt decision tree based on the anatomical region of the medical scan,and further based on the modality of the medical scan. The client devicedisplays via the interactive interface, a plurality of localizationprompts, in succession, beginning with the starting localization prompt,in accordance with corresponding nodes of the localization promptdecision tree until a first one of the set of leaf nodes of thelocalization prompt decision tree is ultimately selected. Theinteractive interface progresses to each next one of the plurality oflocalization prompts by selecting one of the set of branches of eachcorresponding node in accordance with each of a plurality ofcorresponding user localization selections, received via user input.Each of the plurality of corresponding user localization selectionscorresponds to one of the discrete set of selection options for each oneof the plurality of localization prompts displayed via the interactiveinterface.

The client device transmits, via the network, labeling data thatincludes a set of labels indicating the first one of the set of leafnodes of the diagnosis prompt decision tree, the first one of the set ofleaf nodes of the characterization prompt decision tree, and the firstone of the set of leaf nodes of the localization prompt decision tree.

The executable instructions, when executed by the at least one processorof the medical scan hierarchical labeling system, further cause themedical scan hierarchical labeling system to receive, via the network,the set of labels from the client device, and to populate a medical scanentry of the medical scan in the medical scan relational database basedon the set of labels.

FIGS. 13A-13B present embodiments of medical scan annotator system 106,for example, when utilized in conjunction with the medical scanhierarchical labeling system 3002. As illustrated in FIG. 13A, themedical scan annotator system 106 can select a medical scan from themedical scan database 342 for transmission via network 150 to one ormore client devices 120 associated with a selected user set 4010corresponding to one or more users in the user database 344. A medicalscan can be selected for annotation based on an assigned priority and/orbased on a turn-based queue, for example, based on the scan prioritydata 427 of the corresponding medical scan entry 352. The client device120 of each user of the selected user set 4010 can display one or morereceived medical scans to the via the interactive interface 275displayed by a display device corresponding to the client device 120,for example, by displaying medical scan image data 410 in conjunctionwith the medical scan assisted review system 102.

The interactive interface 275 displayed by client devices 120 of eachuser in the selected user set 4010 can include a prompt to provideannotation data 4020 corresponding to the medical scan. This can includea prompt to provide a text and/or voice description via a keyboardand/or microphone associated with the client device. This can alsoinclude a prompt to indicate one or more abnormalities in the medicalscan, for example, by clicking on or outlining a region corresponding toeach abnormality via a mouse and/or touchscreen. For example, theinteractive interface can prompt the user whether or not an abnormalityis present. If the user indicates an abnormality is present, theinteractive interface can prompt the user to identify the region thatincludes the abnormality. This can include allowing the user to scrollthrough one or more slices, to identify one or more slices that containthe abnormality, and to select a region of the one or more slices thatcontains the abnormality. Once the region is identified, the interactiveinterface can prompt the user to provide descriptive informationclassifying an abnormality based on its size, type, etc. To aid the userin providing this information, the user interface can automatically cropone or more slices based on the identified region and/or zoom in on theidentified region. In various embodiments, the medical scan can bepresented for annotation by utilizing the medical scan assisted reviewsystem 102, for example, presented in the new annotation mode. Theinteractive interface 275 can present the medical scan by utilizinginterface features indicated in the display parameter data 470 and/orthe interface preference data 560 of the user, and/or the user canindicate the annotation data via the interactive interface 275 byutilizing interface features indicated in the display parameter data 470and/or the interface preference data 560 of the user. For example, someor all of the annotation data 4020 can correspond to, or beautomatically generated based on, user input to the interactiveinterface.

Annotation data 4020 can be transmitted from each client device of usersin the selected user set 4010 to the medical scan annotator system 106,for example, in response to receiving input data via the interactiveinterface indicating that the annotations are complete. The annotationdata 4020 can be raw annotation data corresponding directly to the userinput, or can be further processed by the client device beforetransmission. For example, a more precise region corresponding to eachabnormality can be determined automatically based on the user input andby determining actual boundary points of the abnormality by utilizingimage processing techniques and/or text and/or voice input can beprocessed and/or parsed, for example, by utilizing a medical scannatural language analysis function and/or medical report analysisfunction to generate medical codes 447 or other diagnosis data 440corresponding to the medical scan. Such processing can also be performedby the medical scan annotation system 106 and/or another subsystem whenthe raw annotation data is received.

The medical scan annotator system 106 can evaluate the set annotationdata 4020 received from the selected user set 4010 to determine if aconsensus is reached, and/or generate a final consensus annotation 4030,for example, by performing an annotation consensus function 4040. Forexample, consider a selected user set 4010 that includes three users. Iftwo users annotate a medical scan as “normal” and the third userannotates the medical scan as “contains abnormality”, the annotationconsensus function 4040 performed by medical scan annotator system 106may determine that the final consensus annotation 4030 is “normal” byfollowing a majority rules strategy. Alternatively, the medical scanannotator system 106 can determine that a consensus is not reachedbecause one of the users indicated that an abnormality is present, andthat the medical scan should not be passed off as normal because a levelof confidence that the scan is normal, determined by a calculatedconsensus confidence score 4050, does not exceed a consensus confidencethreshold. The confidence thresholds required for consensus can differfor different types of scans and/or severity of diagnosis.

If the medical scan annotator system 106 determines that a consensus isachieved, it can automatically generate the final consensus annotation4030, and can map this final consensus annotation to the medical imagein the medical scan database in diagnosis data 440, and/or transmit theconsensus annotation to an originating entity of the medical scan. Themedical scan annotator system 106 can also map the calculated consensusconfidence score to the medical image in the confidence score data 460.In some embodiments, a truth flag 461 will automatically be assigned toall final consensus annotation 4030 in the confidence score data 460and/or will automatically be assigned to final consensus annotation 4030that exceeds a truth threshold. In some embodiments, annotation data4020 received from each user and/or a corresponding annotationconfidence score can also be stored in the medical database, mapped tothe corresponding user and/or the corresponding performance score in theannotation author data 450.

In some embodiments, for example where annotation data 4020 includesseveral attributes, the annotation consensus function 4040 performed bythe medical scan annotation system 106 can determine whether a consensusis reached by calculating a difference between two or more receivedannotation data 4020, for example, by generating a feature vector forannotation data 4020 received from each user. Each feature vector can begenerated based on keywords, medical codes, abnormality location in themedical scan, abnormality size and/or shape in the medical scan, aclassification of the abnormality, or other attributes listed inannotation data 4020 received from each user. Performing the annotationconsensus function 4040 can further include calculating the Euclidiandistance or other vector distance between the two or more featurevectors. Performing the annotation consensus function 4040 can furtherinclude determining if consensus is reached by determining if theaverage of these Euclidian distances is below a certain discrepancythreshold, for example, after determining and removing outlierannotations from the set. Similarly, the annotation consensus function4040 can further include determining if consensus is reached by firstgenerating the final consensus annotation 4030, and then calculating theEuclidian distance between each annotation feature vector and the finalconsensus annotation 4030, where consensus is determined to reached andthe final consensus annotation is confirmed only if the average of thesecalculated Euclidian distances is below a certain discrepancy threshold.The annotation consensus function 4040 can calculate the final consensusannotation 4030 itself by creating a consensus feature vector, whereeach attribute of the consensus feature vector is determined bycalculating a mean, median or mode of each corresponding annotationfeature extracted from all of the received annotation data 4020. In thisfashion, calculating the consensus confidence score 4050 can includecalculating such an average Euclidian distance, where distances withlarger magnitudes correspond to lower or otherwise less favorableconsensus confidence scores 4050, and where distances with smallermagnitudes correspond to higher or otherwise more favorable consensusconfidence scores 4050. Alternatively or in addition, the finalconsensus annotation 4030 can be generated based on the most closelymatching annotations and/or based on another average, for example,calculating an average identified region that includes an abnormality.

The annotation consensus function 4040 further determine whether or notconsensus is reached based on overall or categorized performance scoredata 530 and/or qualification data 540 of each user in the selected userset 4010. For example, each annotation data 4020 can be weighted basedthe performance scores and/or qualifications of the corresponding user.In the example where two users annotate a medical scan as “normal” and athird user annotates a medical scan as “contains abnormality”, themedical scan annotator system 106 may determine that the consensus is“contains abnormality” based on the third user having a much higherperformance score and/or being more highly qualified than the first twousers. The final consensus annotation 4030 can be generated based on theannotation received from a user with the highest ranking in the categorycorresponding to the medical scan. The final consensus annotation 4030can be generated based on calculating a weighted average annotation bycomputing a weighted consensus feature vector, where feature vectors ofhigher ranked users receive a higher weight. In some embodiments, eachfeature of the feature vector can be computed using a different set ofuser weights, for example, where the different feature weights for eachuser is determined based on corresponding category-based performancescore data and/or qualification data.

Alternatively or in addition, the performance score data associated withthe interface features of the interactive interface 275 used by eachuser to annotate the image can also be utilized to weight the differentannotations in reaching consensus. Such weights can be applied whengenerating a consensus feature vector, where each annotation featurevector is weighted according to the performance score data of one ormore corresponding interface features used by the corresponding user.

In some embodiments, confidence scores for each individual annotationcan also be calculated for each user's annotation, and the consensusconfidence score 4050 can be generated based on these confidence scores,for example, based on an average confidence score, based on confidencescores of annotation data that matches the final consensus annotation4030, etc. In some embodiments, the final consensus annotation 4030 canbe generated based on these confidence scores, for example, whereannotation feature vectors are weighted based on a correspondingconfidence score. The confidence scores for each annotation data 4020can be generated automatically, for example, based on performance scoredata 530 as discussed herein. Individual confidence scores and/or aconsensus confidence score 4050 can also be updated retroactively as newannotation data is received, for example, if new annotation data isreceived from another user, for example corresponding to an expertreview when consensus is not reached, and/or if new annotation data isautomatically generated by a subsystem after the consensus data isgenerated.

The medical scan annotator system 106 can also utilize auto-generatedannotation data of the medical scan to determine if consensus is reachedand/or to generate the final consensus annotation 4030. Theauto-generated annotation data can be automatically generated by medicalscan annotator system 106 by utilizing one or more medical scan analysisfunctions. The auto-generated annotation data can also be retrieved fromthe medical scan database 342 if it was generated by a subsystem 101previously. One or more auto-generated annotations can be assigned theirown weights and/or confidence scores, for example, based on the modelaccuracy data 631 and/or another determined performance of the functionand/or subsystem responsible for creating each auto-generatedannotation. Each auto-generated annotation data can be thus treated asan annotation from another user, and can be used to determine ifconsensus is reached and/or to generate the consensus annotation in thesame fashion.

Alternatively, the auto-generated annotation can be merely verifiedbased on the annotation data 4020 received from the selected user set4010 by determining that the user annotations are close enough to theauto-generated annotation based on the discrepancy threshold. Forexample, this process may be utilized by the medical scan diagnosingsystem 108 to perform the output quality assurance step. Theauto-generated annotation can be sent to the selected user set 4010 aspart of this verification process, for example, displayed by eachinteractive interface 275 in conjunction with the medical scan assistedreview system 102 as displayed annotation data, and the annotation data4020 received from the selected user set 4010 can be includeverification of and/or corrections of the auto-generated annotation.Alternatively, the medical scan can be sent without the auto-generatedannotation and/or the auto-generated annotation can be hidden from viewas part of a blind review, to ensure that the users are not biased increating annotation data by the auto-generated annotation.

FIG. 13B illustrates an embodiment of the medical scan annotator system106 upon determining that a consensus is not achieved, for example,because the calculated consensus confidence score 4050 does not exceedthe consensus confidence threshold. The medical scan annotator systemcan select an expert user, for example, a user whose qualification data540 indicates they are an expert in the category corresponding to themedical scan or who otherwise is identified as an expert based on theirperformance score data. The expert can receive the medical scan on acorresponding client device and annotate the image, for example, wherethe interactive interface 275 displays the medical scan image data 410in conjunction with the medical scan assisted review system 102, andwhere the interactive interface utilizes interface features indicated inthe display parameter data 470 of the medical scan and/or indicated inthe interface preference data 560 of the user profile entry 354 of theexpert user. The expert can view the annotation data 4020 generated bythe selected user set 4010, for example, presented as the displayedannotation data of the medical scan assisted review system 102.Annotation data 4020 of each user can be displayed one at a time and theexpert user can elect to advance to the next user's annotation data4020. Alternatively, all of the annotation data 4020 can be displayedsimultaneously for example, in different colors corresponding to eachuser's annotations and/or overlaid as translucent, highlighted regions,for example, where a portion of the highlighted region is more opaquewhen multiple users agree that the portion is included in theabnormality. In other embodiments, the annotation data 4020 can behidden from the expert user, and the expert user can enter their ownannotations in conjunction with a blind review to reduce bias.

Expert annotation data 4070 can be generated automatically, and can betransmitted automatically to the medical scan annotation system 106. Themedical scan annotator system can automatically assign the receivedexpert annotation data 4070 as the final consensus annotation 4030,and/or can assign a truth flag 461 to the expert annotation data 4070 inthe confidence score data 460 of the medical scan. Alternatively, theexpert annotation data 4070 can be compared to the previous annotationdata 4020 and determine if consensus has been reached. For example, theexpert annotation data 4070 and the annotation data 4020 can becollectively utilized by the annotation consensus function 4040, wherethe expert annotation data 4070 is assigned its own, higher weight thanthe other annotations. If consensus has still not been reached, themedical scan annotation system can continue to transmit the image otherusers and processing received annotations until consensus is reached,for example, selecting a new selected user set 4010 and/or selecting anew expert user.

The user profile entries 354 of each user in the selected user set 4010and/or each expert user can be automatically updated by the medical scanannotator system 106 or another subsystem 101 by generating and/orupdating performance score data 530 for each user based comparing theirannotation to the final consensus annotation 4030. For example, theaccuracy score data 531 of the performance score data 530 can begenerated by calculating the Euclidian distance between a feature vectorof a user's annotation and the feature vector of the consensusannotation as described previously, where a higher performance score isassigned to a user whose annotation is a smaller Euclidian distance fromthe consensus, and a lower performance score is assigned to a user whoseannotation is a larger Euclidian distance from the consensus. Theefficiency score data 532 of the performance score data can beautomatically generated, for example, based on an annotation durationdetermined based on a difference between a first time that each userreceived the medical scan and a second time each user completed theannotation. The efficiency score data 532 can be further based on adifference between the annotation duration of each user and an averageannotation duration computed for annotation durations of the selecteduser set. Aggregate performance data for each user can be generateand/or updated based on past accuracy and/or efficiency scores, based onhow many scans have been annotated in total, based on measuredimprovement of the user over time, etc. Similarly, the performance scoredata 630 corresponding to medical scan analysis functions utilized togenerate the auto-generated annotation data can be generated and/orupdated by comparing the auto-generated annotation data to the finalconsensus annotation 4030 in a similar fashion and/or by comparing thecomputed annotation duration of a corresponding medical scan analysisfunctions to other computed annotation durations of other medical scananalysis functions that generated auto-generated annotation data for themedical scan.

The selected user set 4010 can be selected based on the performancescore data 530 and/or qualification data 540 of each user correspondingto previous uses only the medical scan annotation system 106, orcorresponding to usage of several subsystems 101. For example, a medicalprofessional with a user profile indicating that he/she ranks above acertain threshold in annotating CT scans and/or indicating that he/sheis highly qualified in the study of the lungs can be automaticallyselected by the medical scan annotator system to annotate a triagedmedical scan identified as a lug CT scan. The size of the selected userset 4010 that receive a medical scan can be optimized based on thequality of the users selected, for example, based on calculating theprobability of reaching consensus and/or calculating the probabilitythat a consensus confidence score will be above a confidence threshold,and ensuring the probability falls above a probability threshold. Forexample, a first medical scan can be sent to a two medical professionalswith high scores, qualifications, rankings, or correct annotationpercentages. A second medical scan may be sent to ten medicalprofessionals with lower scores or qualifications based on calculatingthat the probability of a correct consensus probability falls above aprobability threshold.

In some embodiments, the medical scan annotator system 106 can firstselect a medical scan for annotation automatically, and in response, theselected user set 4010 can be determined automatically to annotate theselected medical scan based on determining users with highly rankedoverall scores and/or based on categorized performance data 534 and/orqualification data 540 that corresponds to an identified scan classifierdata 420 of the selected medical scan. Alternatively or in addition, theselected user set 4010 can be determined based on the size of a queue ofmedical scans already assigned to each user. For example, the selecteduser set 4010 can correspond to users with matching qualifications thatcorrespond to the scan classifier data 420 and/or correspond to userswith the lowest queues of other medical scans to annotate.

In other embodiments, the medical scan annotator system 106 can firstdetermine one or more available users automatically, for example, basedon medical scan queue lengths for each user in the system and/or inresponse to one or more users requesting to annotate a medical scan. Insuch cases, some or all of these identified users can be added to theselected user set 4010, and the medical scan can be selected based oncorresponding categorized performance data 534, qualification data 540or other relevant user profile data of users in the selected user set4010.

FIGS. 13C-13V present example embodiments of a user interface of amedical scan annotator system 106, for example, presented in conjunctionwith the medical scan assisted review system 102. Some or all featurespresented in FIGS. 13C-13V can also be utilized in conjunction withother subsystems and can be included in the interface features. FIGS.13C-13G present interface features for chest CT nodule characterization,and can be displayed in conjunction with a chest CT scan. Annotationdata 4020 can be generated based on user selections in the userinterface, and can be used to populate abnormality classification data445 for abnormality classifier categories 444 such as “nodulespiculation”, “nodule lobulation”, “nodule texture”, “nodulecalcification”, “nodule sphericity” and/or “nodule internal structure”for the associated medical scan. FIGS. 13H-13J present interfacefeatures for presentation to a user in conjunction with an identifyingchest CT nodule, allowing a user to add new contours for one or morescans for a patient, for example, over multiple years, and indicatemalignancy. As shown in FIG. 13K, the scan can be presented inconjunction with these interface features. FIGS. 13L-130 presentinterface features for presentation to a user in conjunction withidentifying abnormalities in a chest x-ray. Users can classify eachabnormality and draw a shape around each abnormality in the scan.

FIG. 13P presents a view of a chest x-ray presented via the interfacebefore a user identifies regions of interest, and FIG. 13Q presents aview of the chest x-ray via the interface after the user identifiesregions of interest of multiple abnormalities, indicated by sevenpolygons 1022. FIG. 13R presents interface features for comparing chestx-ray severity for multiple patients, displayed in conjunction withmultiple x-rays that can be displayed in adjacent views or can bedisplayed one at a time where the user can toggle between them. A usercan compare multiple scans corresponding to multiple patients, andprovide feedback indicating differences between the patients, comparingif one patient's case is more severe than another, or determine which oftwo scans appears to be more normal.

FIGS. 13S-13V present interface features for chest x-ray triageclassification, displayed in conjunction with a chest x-ray. A user canselect abnormality classification data that can be used to generateannotation data 4020 and/or to populate abnormality classification data445. As shown, some or all abnormality classification categoriesdisplayed, which can be determined based on abnormality classifiercategories 444, can be presented, and hierarchal subcategories can bepresented in response to a user selecting one of a plurality ofabnormality classification categories that are present.

In some embodiments, the medical scan hierarchical labeling system 3002is integrated within and/or utilizes features described in conjunctionwith the medical scan annotator system 106. In particular, theinterfaces presented in some or all of FIGS. 13C-13V can be utilized bythe medical scan hierarchical labeling system 3002, where optionspresented in in some or all of FIGS. 13C-13V can correspond to a set ofselection options of a node in accordance with a prompt decision treeand/or where prompt decision trees of the medical scan hierarchicallabeling system 3002 utilize at least one of the prompts and/or sets ofoptions presented in in some or all of FIGS. 13C-13V as one or moreprompts of one or more prompt decision trees, where the annotation data4020 corresponding to the labeling data generated by the client device.The fixed set of diagnosis, characterization, and/or localizationoptions can include some or all of the selections presented in in someor all of FIGS. 13C-13V, where fields of a medical scan entry correspondto some or all of the selections presented in in some or all of FIGS.13C-13V and/or have valid entries corresponding to some or all of theselections presented in some or all of FIGS. 13C-13V.

The interactive interface can present hierarchical sets of options as auser advances through a prompt decision tree as presented in FIGS.13S-13V, where each set of options is presented as an indented list inaccordance with advancing to a deeper layer of the prompt decision tree,and where each set of options is not presented until the correspondingselection is made. For a hierarchical decision tree corresponding to theprompts presented in FIGS. 13S-13V, labeling data generated in responseto a user selecting “submit and next” at FIG. 13V can indicate a leafnode of a first abnormality, with diagnosis, characterization, andlocalization data fully described as “pulmonary vasculature”,“plethora”, “diffuse”, “left”, and “lobe-left upper”, and can furtherindicate a leaf node of a second abnormality with diagnosis,characterization, and localization data fully described as“mediastinum”, “compression of structure”, and “inferior-anterior”.

FIGS. 14A-14C present an embodiment of a multi-label medical scananalysis system 5002. The multi-label medical scan analysis system canbe operable to train a multi-label model, and/or can utilize themulti-label model to generate inference data for new medical scans,indicating probabilities that each of a set of abnormality classes arepresent in the medical scan. Heat maps for each of the set ofabnormality can be generated based on probability matrices for displayto via a display device.

As shown in FIGS. 14A-14C, the multi-label medical scan analysis system5002 can communicate bi-directionally, via network 150, with the medicalscan database 342 and/or with other databases of the database storagesystem 140, with one or more client devices 120, and/or, while not shownin FIG. 14A, with one or more subsystems 101 of FIG. 1.

In some embodiments, the multi-label medical scan analysis system 5002is an additional subsystem 101 of the medical scan processing system100, implemented by utilizing the subsystem memory device 245, subsystemprocessing device 235, and/or subsystem network interface 265 of FIG.2A. For example, the multi-label medical scan analysis system 5002 canbe implemented by utilizing the medical scan image analysis system 112to train and/or utilize a computer vision model. In some embodiments,the multi-label medical scan analysis system 5002 utilizes, or otherwisecommunicates with, the central server system 2640. For example, themedical scan database 342 can be populated with de-identified datagenerated by the medical picture archive integration system 2600. Themulti-label medical scan analysis system 5002 can receive de-identifiedmedical scans of the training set with their corresponding annotationdata, diagnosis data, and/or medical reports directly from the medicalpicture archive integration system 2600, for example, where theannotation data, diagnosis data, and/or medical reports are utilized todetermine the medical labels for medical scans in the training set. Asanother example, the multi-label medical scan analysis system 5002 canperform an inference function on de-identified medical scans receivedfrom the medical picture archive integration system 2600, andprobability matrix data and/or determined abnormality classes generatedin the inference data can be assigned to the medical scan in the medicalpicture archive integration system 2600. As another example, themulti-label medical scan analysis system 5002 can request de-identifiedmedical scans, annotation data, and/or reports that match requestedcriteria for the training set and/or for new medical scans to belabeled. In some embodiments, some or all of the multi-label medicalscan analysis system 5002 is implemented by utilizing other subsystems101 and/or is operable to perform functions or other operationsdescribed in conjunction with one or more other subsystems 101.

As shown in FIG. 14A, the multi-label medical scan analysis system canbe operable to train a computer vision based model on a plurality ofmedical scans. A medical scan training set can be received from themedical scan database 342 and/or from another subsystem 101. The medicalscan training set can include a plurality of medical scans of the sameor different modality and/or anatomical region. For example, the medicalscan training set can include exclusively chest x-rays. The medical scantraining set can include a plurality of medical labels assigned to theplurality of medical scans. The medical labels assigned to a medicalscan can correspond to at least one of a set of abnormality classes, andeach medical scan in the training set can be labeled with zero, one, ora plurality of labels of the set of abnormality classes that are presentin the medical scan. For example, when the training set includes chestx-rays, the set of abnormality classes can include atelectasis,effusion, mass, pneumonia, consolidation, emphysema, pleural thickening,cardiomegaly, infiltration, nodule, pneumothorax, edema, fibrosis,and/or hernia. The abnormality classes can correspond to some or all ofthe abnormality classifier categories 444, and/or can correspond to anyset of abnormality types or categories, diagnosis types or categories,or medical conditions.

In some embodiments, the training set can further include region ofinterest data for some or all medical scans. The region of interest datacan indicate a portion of the medical scan and/or an anatomical regionwhere the medical label is present in the medical scan, and the modelcan be trained by utilizing this region of interest data. In otherembodiments, no region of interest information is provided, and the atleast one medical label assigned to a medical scan of the training datais considered a global label for the medical scan as a whole.

A training step 5010 can be applied to the medical scan training data togenerate model parameters or other model data corresponding to a trainedmodel. In some embodiments, multiple models are trained by utilizingmultiple sets of training data, for example, where each set of trainingdata corresponds to a different modality and/or anatomical region.Performing training step 5010 can include performing the training step1352 of the medical scan image analysis system 112. In some embodiments,performing the training step 5010 includes training a neural network,for example where the image data of each the plurality of medical scansis the input data. A vector corresponding to the set of abnormalityclasses, populated by binary indicators corresponding to which medicallabels correspond to each of the plurality of medical scans, cancorrespond the output data.

The output data of the training set can alternatively correspond to aN×N×K matrix, where the value of N corresponds to a highest-resolutionlevel of a multi-resolution model, and where the value of K correspondsto number of abnormality classes in the set of abnormality classes. Eachof the N×N values for each of the K classes can correspond to an imagepatch of two-dimensional image data, such as an x-ray or othertwo-dimensional medical image. When no region of interest data isavailable, each of the N×N values for each of the K classes can bepopulated with a binary indicator corresponding to whether thecorresponding abnormality class is present in the image data as a whole,where each of the N×N values for the same K class are assigned to thesame binary indicator. In the case where region of interest data isavailable in training, each of the N×N values for each of the K classescan be populated with a binary indicator corresponding to whether thecorresponding abnormality class is present in the corresponding imagepatch of the image data.

In embodiments where the medical scans of the training set have aplurality of slices, an additional dimension M can correspond to thenumber of slices, where the output data of the training set correspondsto a N×N×M×K matrix, where each of the N×N×M values for each of the Kclasses corresponds to an image patch of the corresponding image slice.In some embodiments, other information of the medical scan accompanyingthe image data can be utilized as input data. For example, patientdemographic and/or history data, and/or other fields of a correspondingmedical scan entry 352 can be utilized to train the model along with theimage data.

The model data generated by performing the training step 5010 cancorrespond to a multi-resolution model such as the model described inFIG. 15A. The model data can be transmitted to other subsystems 101 foruse in implementing an inference function on new medical scans, can betransmitted to medical scan analysis function database 346 for access byone or more other subsystems 101, and/or can be stored locally by themulti-label medical scan analysis system 5002 for use in implementing aninference function on new medical scans.

As shown in FIG. 14B and FIG. 14C, the multi-label medical scan analysissystem 5002 can receive new medical scans from the medical scan database342 and/or from another subsystem 101. The multi-label medical scananalysis system 5002 can perform an inference function 5020 on the newmedical scans by utilizing the model data generated as discussed inconjunction with FIG. 14A. The inference function can be performed onthe image data alone, or can be performed on additional informationalong with the image data, such as metadata of the medical scan, patientdemographic and/or history data, and/or other fields of a correspondingmedical scan entry 352, for example, if the model was trained utilizingsuch information.

The output of the inference function 5020 can include a N×N×K matrixthat indicates N×N probability values for each of the K classes. Each ofthe N×N values, for each of the K classes, can correspond to one of N×Nimage patches of the image data. Each of the N×N probability values foreach of the K classes can correspond to a probability that the acorresponding one of the set of abnormality classes is present in thecorresponding image patch. As discussed in conjunction with FIG. 14A,the value of N can correspond to a resolution of a plurality ofresolution layers of the model. For example, if the size of the imagedata of the new medical scan is 1024×1024 pixels, the value of N cancorrespond to 64 for a total of 64² image patches, each containing 16×16pixels.

In some embodiments, the model data corresponds to a multi-resolutionmodel consisting of multiple resolution layers, where low resolutionlayers capture context and/or semantic information, and high resolutionlayers capture finer details. In particular, the model can include a setof resolution layers, each with a different value n×n×K, where the valueof n×n corresponds to the number of image patches the image ispartitioned into at that resolution layer. Thus, higher values of n cancorrespond to higher-resolution of the layer. The set of resolutionlayers can include an 8×8 layer, a 16×16 layer, a 32×32 layer, and/or a64×64 layer, and the 64×64 layer can correspond to the output layer.

In some embodiments, resolution of the input image data is reduced tolower and lower resolutions, for example, using ResNets. Resolution ateach layer is preserved, for example, using DenseNets, to produce aninitial feature map at each layer. Final feature maps can be generatedat each layer, starting at the lowest resolution layer, and upsamplingthe results to generate the final feature maps at higher resolutionlayers. In particular, resolution preserving nonlinear transformationscan be incrementally applied be on a channel-wise concatenation of aprevious feature map of the current layer and the upsampled feature mapof the previous layer, for example, as discussed in conjunction withFIG. 15A. At the final, highest resolution layer, a probability matrixcan be generated by applying a sigmoid function to the final feature mapof the highest resolution layer, indicating probabilities for eachinstance (at 64×64 resolution), for each of the K classes. Spatialinformation is preserved by utilizing this coarse-to-fine approach,where at lower resolutions, weak localization cues can be furtherrefined in subsequent higher resolutions. In particular, thispreservation of spatial information allows abnormalities to be localizedin inference function 5020 based on the output probability matrix, evenin embodiments where no region of interest data was utilized in trainingstep 5010.

In embodiments where the medical scans of the training set have aplurality of slices, an additional dimension M can correspond to thenumber of slices, where the output data of the training set correspondsto a N×N×M×K matrix, where each of the N×N×M values for each of the Kclasses corresponds to an image patch of the corresponding image slice.The value of M can be constant across each of the plurality ofresolution layers of the model, where there is a plurality of n×n imagepatches for each of the M slices at each of the resolution layers. Inother embodiments, the plurality of image patches are three-dimensionalimage subregions at some or all of the resolution layers, where thethree-dimensional image subregions include pixels from multiple imageslices. The slice-wise resolution can also improve with each resolutionlayer, for example, where the number of slices in each three-dimensionalimage subregion decreases at higher resolution layers. The number ofslices in each three-dimensional image subregion at the highestresolution layer can be equal to one, or can be equal to a differentlowest number.

In some embodiments, performing the inference function 5020 can includeperforming the inference step 1354 of FIG. 7B to generate one or moreprobability matrices 1371, with an added dimension K corresponding toeach of the plurality of abnormality classes. Thus, the probabilitymatrices 1371 can be generated for each of the K abnormality classes.The detection step 1372, abnormality classification step 1374, similarscan identification step 1376, and/or display parameter step 1378 cansimilarly be performed separately for each of the K abnormality classes.

The probability matrix data generated as output of the inferencefunction 5020 can be transmitted to the client device 120 for displayvia a display device. Alternatively or in addition, the probabilitymatrix data can be transmitted for use by another subsystem 101 and/orcan be transmitted to the medical scan database to be mapped to thecorresponding medical scan. Alternatively or in addition, theprobability matrix data can be utilized by the multi-label medical scananalysis system 5002 to generate saliency maps, to generate region ofinterest data, to generate global probabilities for each class asillustrated in FIG. 14B, and/or to generate heat map visualization dataas illustrated in FIG. 14C.

Generating the saliency maps can include assigning each value in theprobability matrix to 1 or 0, and can be generated for each resolutionlevel. Each value in the probability matrix can be assigned the value 1or 0 by comparing the raw probability to a threshold. The threshold canbe the same or different for each of the K abnormality classes, and/orcan be the same or different for each of the N×N image patch locations.The saliency maps can be visually presented, for example, where imagepatches assigned a value of 1 are displayed in white or another firstcolor, and where image patches assigned a value of 0 are displayed inblack or another second color. The saliency maps can be transmitted tothe client device 120 for display via a display device. Alternatively orin addition, the saliency maps can be transmitted for use by anothersubsystem 101 and/or can be transmitted to the medical scan database tobe mapped to the corresponding medical scan.

Generating the region of interest data can include comparing values ofeach probability matrix to a probability threshold. The probabilitythreshold can be the same or different for the plurality of abnormalityclasses. In some embodiments, the region of interest data indicates oneor more image patches with a probability value that compared favorablyto the probability threshold for at least one of the K abnormalityclasses. In some embodiments, the at least one of the K abnormalityclasses for which the probability value compared favorably to theprobability threshold is further indicated in the region of interestdata. The region of interest data can be transmitted for use by anothersubsystem 101 and/or can be transmitted to the medical scan database tobe mapped to the corresponding medical scan. Alternatively or inaddition, the region of interest data can be transmitted to the clientdevice 120 for display via a display device. In particular, the regionof interest data can be displayed in conjunction with the medical scan,for example, where the one or more image patches identified in theregion of interest data are outlined, highlighted, or otherwiseindicated visually. Furthermore, the at least one of the abnormalityclasses corresponding to the region of interest can be identified astext, and/or as a color or pattern used to identify the region interest.For example, for region of interest data of a medical scan with a firstregion of interest corresponding to a first abnormality class and asecond region of interest corresponding to a second abnormality class,the first region of interest can be indicated via an interface displayedby the display device by overlaying the corresponding image patch of theimage data of the medical scan with a first color and/or pattern, andthe second region of interest can be indicated via the interface byoverlaying the corresponding image patch of the image data of themedical scan with a second color and/or pattern. In some embodiments,the region of interest data is presented by utilizing the interface ofthe medical scan assisted review system 102. In some embodiments, theregion of interest data is presented by the interface of a multi-labelheat map display system.

FIG. 14B illustrates an embodiment of a multi-label medical scananalysis system 5002 that is implemented as a global multi-labelgenerating system. The multi-label medical scan analysis system 5002 cangenerate global probabilities, for example, by performing a globallabeling function 5030 on the probability matrix data. Generating theglobal probabilities can include evaluating the N×N probability matrixfor each of the K classes and determining a global probability value foreach of the K classes. In some embodiments, the global probability valuefor a given abnormality class is assigned a highest probability of thecorresponding N×N probability matrix. In some embodiments, the globalprobability value for an abnormality class is based on an average ofsome or all of the probability values of the corresponding N×Nprobability matrix, and/or based on some other function of the values ofthe N×N probability matrix. In some embodiments, determining the globalprobability value includes applying a low-pass filter or other filter tothe probability matrix.

In some embodiments, a final binary identifier is assigned for each ofthe K classes, indicating whether each of the K abnormality classes aredetermined to be present or absent in the medical scan, based on theirglobal probabilities. For example, each of the K global probabilitiescan be compared to a probability threshold, which can be the same ordifferent for each of the K abnormality classes. The abnormality can bedetermined to be present when the corresponding global probability valuecompares favorably to the probability threshold, and can be determinedto be absent when the corresponding probability value comparesunfavorably to the probability threshold.

The global probability data and/or final binary identifiers determinedfor some or all of the K classes can be transmitted to the client device120 for display via a display device. Alternatively or in addition, theglobal probability data and/or final binary identifiers are transmittedfor use by another subsystem 101 and/or are transmitted to the medicalscan database to be mapped to the corresponding medical scan. In someembodiments a single, global binary identifier is generated,alternatively or in addition for the final binary identifiers for eachof the K classes. The global binary identifier can be generated based onthe probability matrix data, the global probability values, and/or finalbinary identifiers, and can indicate whether or not the medical scan isdetermined to include any abnormalities.

In some embodiments, the K abnormality classes are treated asindependent variables. In such cases, the global probability for eachabnormality class can be computed based only on corresponding the N×Nmatrix, independent of other N×N probability matrixes of the other K−1abnormality classes. Similarly, the final binary identifier for each ofthe abnormality classes can be computed based only on the correspondingglobal probability, independent of the global probabilities of the otherK−1 abnormality classes.

In other embodiments, dependency of some or all of the K abnormalityclasses is utilized in computing the global probabilities and/or thefinal binary identifiers. For example, correlation data that indicatescorrelations between pairs and/or subsets of the K abnormality classescan be determined and/or learned based on the training data, and theglobal probabilities and/or binary identifiers can be generated based onthe correlation data. For example, the global probabilities can becomputed as joint probabilities utilizing the correlation data. Asanother example, a global probability value for a first abnormalityclass can be set to a higher value in response to a high globalprobability value being computed for a second abnormality class that hasa high correlation with the first abnormality class. In someembodiments, the correlations are learned and inherently integratedwithin the model and are reflected when the probability matrices aregenerated as output when the inference function is performed, andfurther consideration of abnormality class interdependencies is notnecessary.

In various embodiments, a multi-label medical scan analysis system 5002implemented as a global multi-label generating system is operable toreceive, via a receiver, a plurality of medical scans and a plurality ofglobal labels corresponding to the plurality of medical scans. Each ofthe plurality of global labels corresponds to one of a set ofabnormality classes. A computer vision model is generated by training onthe plurality of medical scans and the plurality of global labels.Probability matrix data is generated by performing an inference functionthat utilizes the computer vision model on a new medical scan. Theprobability matrix data includes, for each of a set of image patches ofthe new medical scan, a set of patch probability values corresponding tothe set of abnormality classes. Each of the set of patch probabilityvalues indicates a probability that a corresponding one of the set ofabnormality classes is present in the each of the set of image patches.Global probability data is generated based on the probability matrixdata. The global probability data indicates a set of global probabilityvalues corresponding to the set of abnormality classes, and each of theset of global probability values indicates a probability that acorresponding one of the set of abnormality classes is present in thenew medical scan. The global probability data is transmitted, via atransmitter, to a client device for display via a display device.

FIG. 14C illustrates an embodiment of the multi-label medical scananalysis system 5002 implemented as a multi-label heat map generatingsystem. Heat map visualization data is generated, for example, byperforming a heat map generator function 5040 on the probability matrixdata. In particular, the heat map visualization data can indicate a heatmap for each of the K abnormality classes, based on their correspondingN×N probability matrix. Each heat map can indicate pixel values or othercolor values, corresponding to grayscale and/or RGB color values,corresponding to each pixel of the input image data. In someembodiments, pixel values and/or other color values are only indicatedfor each image patch of the highest resolution, for example, where asingle color value is computed for each of the values of the N×Nprobability matrix for each of the K classes. In some embodiments, heatmaps are generated for each of the resolution layers, for example, wherea single color value is computed for each image patch of thecorresponding resolution layer.

The pixel values of each heat map can be proportional to the rawprobability values of the N×N probability and/or can be computed as adeterministic function of the raw probability values. This results in aN×N resolution heat map, and if displayed by the same number of pixelsas the input image, results in all pixels of the same image patch beingassigned the same color. This can result in borders between imagepatches having a dramatic shift in color. For a more visually desirableheat map, a smoothing function can be utilized to smooth the colortransitions between image patches and/or to otherwise soften the bordersbetween image patches by changing color values gradually within eachimage patch in the direction towards each of up to four borders, basedon the color of the neighboring image patch in each of the up to fourdirections. For example, when a 64×64 dimension probability matrix isoutputted for a 1024×1024 image for each abnormality class, smoothingtechniques can be applied within each of the 16×16 dimension patches,for example, to smooth the borders between patches. In this case, thesame image patch can include pixels of varying colors. Color valuedifferentials between ones of a set of initial color values ofneighboring pixels included in different ones of the set of imagepatches can be reduced as a result of applying the smoothing function.In some embodiments, the smoothing function is different for some or allof the K abnormality classes.

Alternatively or in addition, a segmentation masking function can beapplied to some or all of the heat maps to mask one or more designatedor determined regions, for example, based on borders of an anatomicalregion. For example, the segmentation masking function can maskeverything outside of the heart. Masking can include not assigning pixelvalues for the masked region of the heat map and/or can include settingthe pixel values for the masked region of the heat map to a mask colorsuch as black, white, or other uniform, predetermined mask color. Insome embodiments, the segmentation masking function is different forheat maps of each of the K different abnormality classes, for example,based on predetermined anatomical regions the different ones of the Kabnormality classes pertain to. For example, for chest x-rays, heat mapsfor some of the K abnormality classes can mask regions outside thelungs, for example, when the corresponding abnormality class isprevalent in the lungs. As a further example, other ones of the Kabnormality classes can mask regions outside the heart, for example,when the corresponding abnormality class is prevalent in the heart.

This heat map visualization data can be transmitted to a client devicefor display via a display device. In some embodiments, an interfacedisplayed by the display device displays at least one of the heat mapsfor a corresponding one of the K abnormality classes. Each heat map canbe displayed based on the corresponding color values for each pixel. Insome embodiments, the raw pixel values of the heat map are displayed. Insome embodiments, the heat map overlays the medical scan image data,where features of the original image data are still visible and the heatmap highlights or shades the original image data in accordance with thecolor values of the heat map visualization data. The heat maps can bedisplayed in conjunction with text identifying the correspondingabnormality class, with the calculated global probability value for thecorresponding abnormality class, and/or with the final binary identifierof the corresponding abnormality class. In some embodiments, theinterface described in conjunction with the medical scan assisted reviewsystem 102 is utilized to display the heat maps and/or to allow a userto interact with the interface.

In various embodiments, a multi-label medical scan analysis system 5002implemented as a multi-label heat map generating system is operablereceiver, via a receiver, a plurality of medical scans and a pluralityof global labels corresponding to the plurality of medical scans, whereeach of the plurality of global labels correspond to one of a set ofabnormality classes. A computer vision model can be generated bytraining on the plurality of medical scans and the plurality of globallabels. A new medical scan can be received, via the receiver.Probability matrix data can be generated by performing an inferencefunction that utilizes the computer vision model on the new medicalscan. The probability matrix data includes, for each of a set of imagepatches of the new medical scan, a set of patch probability valuescorresponding to the set of abnormality classes. Each of the set ofpatch probability values indicates a probability that a correspondingone of the set of abnormality classes is present in the each of the setof image patches. Heat map visualization data can be generated based onthe probability matrix data. The heat map visualization data canindicate, for each of the set of abnormality classes, a color value foreach pixel of the new medical scan. The heat map visualization data canbe transmitted to a client device for display via a display device.

The discussion that follows in conjunction with FIGS. 15A-15D introducesan example embodiment of the model generated by multi-label medical scananalysis system 5002. Diagnostic imaging often requires the simultaneousidentification of a multitude of findings of varied size and appearance.Beyond global indication of said findings, the prediction and display oflocalization information improves trust in and understanding of resultswhen augmenting clinical workflow. Medical training data rarely includesmore than global image-level labels as segmentations are time-consumingand expensive to collect. The example embodiment of the multi-labelmedical scan analysis system 5002 utilizes a novel architecture, whichlearns at multiple resolutions while generating saliency maps with weaksupervision.

As used herein, x∈

^(w×h×c) denotes an input image with width w, height h, and channel c.In particular, x can correspond to the medical scan received by themulti-label medical scan analysis system 5002. As used herein, y is abinary vector of dimensionality K, where K is the total number ofclasses. In particular, K can correspond to the size of the set ofabnormality classes discussed herein. For a specific class k, y_(k)=0indicates its absence and y_(k)=1 its presence. The subscript indexes aparticular example, for instance, {x_(i); y_(i)} is the i-th example. Asused herein, F∈

^(w×h×c) denotes a feature map and Q∈

^(w×h×K) denotes a saliency map, with Q∈[0,1]. As used herein, two depthfactors l and m accompany the feature and saliency maps. For instance,F^(l) is the feature map as the result of a set of nonlineartransformation that changes the spatial resolution of F^(l−1). On theother hand, F_(m−1) and F_(m) are consecutive feature maps that preservethe resolution during the nonlinear transformation.

FIG. 15A illustrates an example model that can be utilized by the of themulti-label medical scan analysis system 5002. In particular, FIG. 15Aillustrates an example inference function 5020 that produces a saliencymap with a resolution of 64×64, illustrating the process from inputX-ray image to a predicted abnormality score. To reduce the resolution,a standard ResNet is firstly applied on the input image. To preserve theresolution, a standard DenseNet is applied per resolution. Upsamplingand channel-wise concatenation fuse information from multipleresolutions. LSE-LBA pooling aggregates instance scores to the globalprobability Different numbers of resolution layers and/or differentresolutions at each layer can be utilized in other embodiments. In someembodiments, the inference function 5020 and/or the global labelingfunction 5030 can be implemented by utilizing the model of FIG. 15A asdiscussed herein.

Each ResNet can contain several sub-modules, each of which isparameterized as F^(l+1)=σ(g(F^(l))+ƒ(F^(l))). F^(l+1) can be half theresolution of F^(l), and/or can have twice the number of channels asF^(l). σ can be is an element-wise nonlinearity. The functions g and ƒcan be composed of a series of 1×1 and 3×3 convolutions. The reductionin spatial resolution can be achieved by using convolutions with astride size 2. A simple ƒ and complex g can be chosen such that ƒ is asclose as possible to a simple identity transformation, leaving theheavy-lifting non-linear transformations to g to learn the residual. Insome embodiments, spatial resolutions can be preserved withF_(m+1)=σ(g(F_(m))+F_(m)) in which case ƒ is chosen to be the identityfunction.

Because ResNets are susceptible to over-parameterization, which becomescritical when residual connections are used repeatedly on the horizontaldata row in FIG. 15A without changing the spatial resolution. In thescenario where F^(l+1)=σ(g(F^(l))+ƒ(F^(l))) is applied repeatedly, amodel could simply learn to ignore the capacity in g, especially when σis a rectified linear unit (relu). This would effectively defeat thepurpose of inner-resolution propagation where a model is encouraged tospecialize in making predictions under a selected resolution l. To solvethis issue, the non-identity transformation on F^(l) can be enforcedexplicitly, which can include removing the residual connections. Becausethe resulting model would lose the attraction of being easy to optimize,DenseNets can be utilized, where the resolution-preservingtransformation is formulated as F_(m)=σ(ƒ(F₁⊕F₂⊕ . . . ⊕F_(m))), where ⊕denotes the channel-wise concatenation of feature maps and ƒ denotes aseries of resolution-preserving nonlinear transformations. This equationfor F_(m) enforces the nonlinear transformation ƒ on all previousfeature maps without the possibility of skipping using identity mappingwhile still maintaining the desirable property of being easy to optimizedue to the direct connections with all previous feature maps. Such adesign effectively encourages the participation of all previous featuremaps in propagation.

Fine-scale features, computed at high resolutions, capture detailedappearance information while coarse-scale features, computed from lowerresolution representations of the data, capture semantic information andcontext. In deep neural networks utilized by the multi-label medicalscan analysis system 5002, fine-scale features are learned in theearliest layers and coarse-scale features are learned in the subsequentlayers, where the spatial resolution of the data has been reduced byrepeated downsampling operations. Thus, the model learns to construct afeature hierarchy in a fine-to-coarse manner. While the coarse-scalefeatures at the top of typical classification neural networks aresuitable for image-level classification, spatial information required toprecisely localize abnormalities is likely to be lost. If the model isexpected to predict not only what abnormalities are present in the imagebut where they are, then the spatial information must be reintegrated.

The model illustrated in FIG. 15A performs this incrementally, in acoarse-to-fine manner, by repeatedly performing the operation F_(m)^(l)=ƒ(

(F_(n) ^(l+1))⊕(F_(m−1) ^(l))), where F_(m) ^(l) denotes the m-thresolution-preserving feature map at resolution level l, where F_(n)^(l+1) denotes the n-th feature map from the lower resolution level l+1,where F_(m−1) ^(l) denotes previous feature map at resolution level l,where

denotes the upsampling operation, and where ⊕ denotes the channel-wiseconcatenation. The upsampling operation

, can be implemented in various ways including bilinear interpolation,nearest-neighbors interpolation, and/or learnable transposedconvolutions. In the example embodiment discussed here,nearest-neighbors can be used to implement

.

Log-Sum-Exp Pooling with Lower-bounded Adaptation can be utilized totake a saliency map S of a particular class k and produces a final scorep, and can be defined as follows:

$p = {{{LSE}\text{-}{{LBA}(S)}} = {\frac{1}{r_{0} + {\exp(\beta)}}\log\left\{ {\frac{1}{wh}{\sum\limits_{i = 1}^{w}{\sum\limits_{j = 1}^{h}{\exp\left\lbrack {\left( {r_{0} + {\exp(\beta)}} \right)S_{i,j}} \right\rbrack}}}} \right\}}}$

As used herein, S∈Q^(w×h×1) denotes a two-dimensional saliency map for aparticular class k to be pooled. S_(i,j) denotes the (i, j)-th elementof S. In other embodiments, a Noisy-OR (NOR) function, generalized-mean(GM) function, and/or Log-Sum-Exponent (LSE) function can be utilized inperforming the pooling function to generate the final score p. The finalscore p can correspond to the global probability determined for thecorresponding abnormality class k as discussed in conjunction with FIG.14B. For example, the global labeling function 5030 can be implementedby utilizing the LSE-LBA function, and/or another pooling function.

In addition to maintaining the benefits of using a pooling functionwhich balances average and max pooling, the LBE-LSA pooling function isrobust to the issue of numerical underflow when S_(i,j) is very close tozero, compared with other pooling functions such as NOR and GM pooling,due to the removal of the exponential that directly acts on S_(i,j).LSE-LBA also preserves probabilities. By bounding the values in S to bein the range [0; 1], the resulting score will also be in the sameinterval. Since the LSE-LBA function is monotonically increasing inS_(i,j), it attains its maximum value when all S_(i,j)=1, and itsminimum value when all S_(i,j)=0. When S is a map of all O's,LSE-LBA(S)=0 and, and when S is a map of all l's LSE-LBA(S)=1. A sigmoidactivation function can be used on each S_(i,j) to maintain thisproperty. In addition to being numerically stable in computation, ourthe LSE-LBA function reparametrizes a hyperparameter r, used in LSEpooling, with r=r₀+exp(β) where r₀ is a positive constant and β alearnable parameter. r can be lower bounded by r₀, expressing thesharpness prior of the pooling function. A large r₀ can encourage thelearned saliency map to have less diffuse modes.

The model of FIG. 15A can be utilized in a weakly-supervised settingwhere pixel-wise labels are not available and only image-levelannotations are utilized. Given the multi-resolution fused feature mapat the highest level resolution F⁰∈

^(w×h×c), it is further divided into a grid of N×N, with N being thechosen resolution of the final saliency map. In some embodiments, N=w=h,resulting in F⁰∈

^(N×N×c). Each of the N² c-dimensional vectors represents an instanceI_(n)(x) in the bag F⁰, where n={1, . . . N²}. The K-class instanceprobability is P(I_(n)(x))=sigmoid(WI_(n)(x)), where W is a K by cparameter matrix that is shared among all N² instances. This leads tothe final probabilistic saliency map S∈Q^(N×N×K). Following the LBE-LSApooling function, P(x)=LSE-LBA(S(x)), where prediction P (x) is aK-dimensional vector and represents, according to theprobability-preserving property of LSE-LBA pooling the probability of xbelonging to K classes. Hence, a multi-class cross-entropy cost can bedirectly computed given y.

FIG. 15B illustrates example output saliency maps at various resolutionsfor an input chest x-ray. The model can produce all of the saliencymaps, corresponding to each of the resolution levels, or can produce thehighest level saliency map only. The model of FIG. 15A can also beutilized to generate probability matrix data and/or global probabilitydata for any other types of medical scans described herein.

The model of FIG. 15A can be applied to datasets of medical scans, suchas the NIH Chest X-ray dataset, which contains 112,120 frontal-viewchest X-rays taken from 30,805 patients, where 51,708 images contain atleast one of 14 labeled pathologies, in a PNG format with a standardizedspatial resolution of 1024×1024. The 14 labeled pathologies cancorrespond to the set of abnormality classes, and can include, forexample, atelectasis, effusion, mass, pneumonia, consolidation,emphysema, pleural thickening, cardiomegaly, infiltration, nodule,pneumothorax, edema, fibrosis, and/or hernia. Other clinical informationincluding patients' age and gender are accessible in addition to thepathology labels, and while not used in this example embodiment, can beutilized in other example embodiments to train the model and/orimplement the inference function.

For computational efficiency, the inputs of 1024×1024 can be downsampledto 512×512. Data augmentation can be applied during training, forexample, where each image is zoomed by a factor uniformly sampled from[0.25; 0.75], translated in four directions by a factor uniformlysampled from [−50; 50] pixels, and/or rotated by a factor uniformlysampled from [−25; 25] degrees. After data augmentation, the inputs canbe normalized to the interval [0; 1]. To further regularize the model, aweight decay can be applied, for example, with a coefficient of 10⁻⁵.

The model can be trained from scratch using only the NIH training set,for example, with an Adam optimizer and a learning rate of 0.001. Earlystopping can be performed on the validation set based on the average AUC(Area Under the ROC curve) over all of the set of pathologies. Forclassification, the AUC per abnormality can be utilized.

In some embodiments, no bounding boxes are used at training time so thatthe model remains weakly supervised with respect to the task oflocalization. The best models on the classification task can then beevaluated on their localization performance. The quality of localizationcan be determined using the metric of intersection over detectedbounding boxes (IoBB) with T(IoBB)=α, where α is set at a certainthreshold. IoBB can be extremely sensitive to the choice of thediscretization threshold by which the predicted probability score S isbinarized before being compared with ground truth bounding boxes. IoBBcan be very sensitive to the choice of a binarization threshold τ todiscretize probabilistic saliency maps into binary foreground andbackground masks.

In some embodiments, the continuous version of DICE=(2×S×G)/(S²+G²) canbe utilized as the cost function for training the model as a semanticsegmentation model, where S is the probabilistic saliency map directlyoutput by the model, and where G the ground truth binary bounding boxdownsampled to 512×512, the same resolution as the model input. The DICEcost function, or another cost function, can be selected to take intoaccount the probability while avoiding the decision of having to selectthe discretization threshold τ.

FIG. 15B presents a table illustrating an example of abnormalityclassification and weakly supervised localization performance on 14abnormalities on the NIH Chest X-ray test set. Three models withdifferent lower-bounded adaptation r₀ are included. In some embodiments,the impact of r₀ is much more pronounced in localization than inclassification. In this example, the model is only trained on NIH data.In other embodiments, a pre-trained model, for example, trained onImageNet without multi-resolution fusion. The bolded numbers of thetable of FIG. 15B indicate the maxima other than statisticalsignificance. Compared with classification, the choice of r₀ can have amore significant impact on abnormality localization due to their likelydistinct visual appearance. For instance, when r₀ is small and thesharpness prior is weak, a model can tend to perform well on visuallydiffused abnormalities such as cardiomegaly, infiltration and pneumonia.As the sharpness prior is strengthened, localization of focalized andpatchy abnormalities can be improved, as in the case of atelectasis andnodule. When choosing r₀ to be large, the performance of diffusedabnormalities can degrade, such as atelectasis, cardiomegaly, effusionand pneumonia.

FIG. 15C includes an example of model-generated saliency maps. Inparticular, for each of the abnormality classes cardiomegaly,infiltration, nodule, effusion, mass, and pneumonia, FIG. 15C includes,from left to right, original images, ground truth bounding boxes, andmodel generated saliency maps for each of r₀=0, r₀=5, r₀=10,respectively. The corresponding DICE score for each model-generatedsaliency map, computed with respect to the ground truth, is alsopresented above the corresponding saliency map. FIG. 15C illustratesthat increasing r₀ can result in overall sharper saliency maps. Usingbounding boxes to delineate abnormalities can be limited byover-estimating their true ROIs, which is illustrated in the cases ofinfiltration and pneumonia. As illustrated in FIG. 15C, some modelfindings can be incorrectly marked as false positives due to labelingnoise where the ground-truth reader missed the finding.

FIG. 15D illustrates another example of saliency maps at multipleresolutions generated for a chest x-ray with mass by utilizing the fourmodels, with an increasing target resolution, that were trained, forexample, as discussed in conjunction with FIGS. 15A-15C, to produce thepresented visualization. In particular, FIG. 15D illustrates howmulti-resolution, lower-resolution maps can provide weak localizationcues that are refined in higher-resolution layers. In some embodiments,only a highest resolution saliency map, such as a 64×64 resolutionsaliency map, is generated for an input medical scan.

FIGS. 16A-16E illustrate embodiments of a multi-model medical scananalysis system 6002. Multiple models can be trained to process medicalscans of different views of a patient, medical scans of differentmodalities, and/or medical scans of different anatomical regions.Alternatively or in addition, some of the multiple models can be trainedto detect/characterize different types of abnormality patterns and/ordifferent characteristics of various types of abnormalities. When newmedical scans are received for processing, one or more models can beselected based on the new medical scan, and inference data for the newmedical scan can be generated by performing the one or more selectedmodels. Inference data thus generated for new medical scans by selectinga proper subset of the set of models based on features of the newmedical scans that correspond to criteria utilized to train the propersubset of models. This illustrates an improvement over existing systems,allowing more precise inference data to be generated by utilizingparticular models trained to process the type of scan presented and/ortrained to process a particular type of detected abnormality in the typeof scan presented.

As shown in FIGS. 16A-16E, the multi-model medical scan analysis system6002 can communicate bi-directionally, via network 150, with the medicalscan database 342 and/or with other databases of the database storagesystem 140, with one or more client devices 120, and/or, while not shownin FIGS. 16A-16E, with one or more subsystems 101 of FIG. 1, with thecentral server system 2640, and/or with the medical picture archivesystem 2600.

In some embodiments, the multi-model medical scan analysis system 6002is an additional subsystem 101 of the medical scan processing system100, implemented by utilizing the subsystem memory device 245, subsystemprocessing device 235, and/or subsystem network interface 265 of FIG.2A. For example, the multi-model medical scan analysis system 6002 canbe implemented by utilizing the medical scan diagnosing system 108, themedical scan image analysis system 112, and/or the multi-label medicalscan analysis system 5002 to train the plurality of computer visionmodels and/or to perform a plurality of inference functions by utilizingthe plurality of computer vision models.

In some embodiments, the multi-model medical scan analysis system 6002utilizes, or otherwise communicates with, the central server system2640. For example, the medical scan database 342 can be populated withde-identified data generated by the medical picture archive integrationsystem 2600. The multi-model medical scan analysis system 6002 canreceive de-identified medical scans of the training set with theircorresponding annotation data, diagnosis data, and/or medical reportsdirectly from the medical picture archive integration system 2600, forexample, where the annotation data, diagnosis data, and/or medicalreports are utilized to determine the medical labels for medical scansin the training set. As another example, the multi-model medical scananalysis system 6002 can perform one or more inference functions onde-identified medical scans received from the medical picture archiveintegration system 2600, and abnormality data or other inference datagenerated as output of the plurality of the one or more inferencefunctions can be assigned to the medical scan in the medical picturearchive integration system 2600. As another example, the multi-modelmedical scan analysis system 6002 can request de-identified medicalscans, annotation data, and/or reports that match requested criteria forthe training set and/or for new medical scans to be labeled. As anotherexample, one or more of the inference functions trained by themulti-model medical scan analysis system 6002 can be utilized by theannotating system 2612. In some embodiments, some or all of themulti-model medical scan analysis system 6002 is implemented byutilizing other subsystems 101 and/or is operable to perform functionsor other operations described in conjunction with one or more othersubsystems 101.

As shown in FIG. 16A, a training set of medical scans can be received,for example, from the medical scan database 342. Some or all medicalscans in the training set of medical scans can include image data.Alternatively or in addition, some or all medical scans in the trainingset of medical scans can include raw sensor data captured by an modalitymachine, for example, prior to performing a Fourier transform togenerate the image data of the medical scan. Alternatively or inaddition, some or all medical scans in the training set of medical scanscan include some or all of the fields of a corresponding medical scanentry 352. Alternatively or in addition, some or all medical scans inthe training set of medical scans can correspond to DICOM images thatincludes a DICOM header. Alternatively or in addition, output labelingdata that indicate and/or characterize at least one abnormalitydetermined to be present in the medical scan can be received inconjunction with the corresponding medical scan for some or all medicalscans in the medical scan training set.

A grouping step 6005 can be performed on the medical scan training setto form a plurality of training subsets 1-K. Some or all of the trainingsubsets can have non-null intersections. Alternatively, training subsets1-K can be mutually exclusive. Training subsets 1-K can be collectivelyexhaustive with respect to medical scans in the medical scan trainingset, or one or more medical scans of the medical scan training set canmay be included in none of the training subsets 1-K. One or more of thetraining subsets 1-K can include all of the medical scans in the medicalscan training set. One or more training subsets 1-K can include the sameset of medical scans from the medical scan training set, but can betrained based on different criteria, for example, to detect differenttypes of abnormalities.

The grouping step 6005 can include assigning the medical scans to thetraining subsets 1-K in accordance with grouping criteria, where medicalscans that meet grouping criteria for one or more of the trainingsubsets are included in the one or more training subsets. The groupingcriteria can be based on characteristics of the scan itself, for exampleby utilizing some or all of scan classifier data 420, where medicalscans are grouped by modality, anatomical region, view, sequence,originating entity, geographic region, date, and/or other dataclassifying the medical scan. Alternatively or in addition, the groupingcriteria can be based on some or all of diagnosis data 440, for example,where medical scans are grouped by abnormality location data 443 and/orwhere medical scans are grouped by abnormality classifier category 444and/or by abnormality pattern category 446. For example, groupingcriteria for one of the training subsets can require medical scans thatare Head CTs from a particular geographic region with abnormality dataindicating a brain tumor. The corresponding sub-model can be trained tomore accurately detect or to further characterize brain tumors forpatients from the geographic region based on corresponding labeling datafor the medical scans in this training subset.

Some or all of a medical scan's characteristics utilized to determinewhich group the medical scan is assigned to can be determinedautomatically by applying existing inference functions to generate thecorresponding information for the medical scans. For example, modality,sequence, anatomical regions, and/or diagnosis data can be automaticallydetermined by utilizing an existing computer vision model on the imagedata of the medical scan, such as the input quality assurance function1106, and grouping medical scans in accordance with the determined scancategory 1120.

Any fields, such as fields with discrete sets of options, of medicalscan entries 352 can be utilized to group the medical scans, where onlymedical scans of the training set with one or more fields of a medicalscan entry 352 that match and/or compare favorably to requirements forthe one or more corresponding fields in the grouping criteria for one ofthe training subsets 1-K will be included in the one of the trainingsubsets 1-K. In some embodiments, a discrete set of standardized fieldsin standardized DICOM headers or other metadata of the set of medicalscans can be utilized to group the medical scans, where only medicalscans of the training set with one or more DICOM fields of that matchand/or compare favorably to requirements for the one or morecorresponding fields in the grouping criteria for one of the trainingsubsets 1-K are included in the one of the training subsets 1-K.

In some embodiments, a discrete set of standardized output labels areutilized to group the medical scans. This can include training onlabeling data generated by user input to an interactive interface 3075by utilizing the medical scan hierarchical labeling system 3002, and/orannotation data and/or additional annotation data generated by utilizingmodel-assisted annotating system of FIGS. 18A-18E, can be utilized togroup the medical scans. For example, only medical scans of the trainingset with one or more output labels that match and/or compare favorablyto requirements for the one or more corresponding output labels in thegrouping criteria for one of the training subsets 1-K are included inthe one of the training subsets 1-K. In some embodiments, the medicalscans are grouped in accordance with internal nodes and/or leaf nodes ofone or more prompt decision trees of the medical scan hierarchicallabeling system. In some embodiments, medical scans with output labelingdata that corresponds to a leaf node branching from a particularinternal node of a prompt decision trees are grouped in the sametraining subset, where some or all training subsets correspond todifferent internal nodes of one or more prompt decision trees.

In some embodiments, performing the grouping step 6005 includingpartitioning data of at least one of the medical scans in the trainingset of medical scans. For example, in an embodiment where the groupingstep groups different sequences, a medical scan can be partitioned intoa plurality of different sequences, where each sequence is included adifferent one of the set of training subsets. As another example, in anembodiment where the grouping step groups different anatomicalsubregions, image data of a medical scan can be partitioned into aplurality of portions by different anatomical subregions, where each ofthe partitioned portions are included in a different one of the set oftraining subsets. In particular, different subsets of a set of imageslices included in the medical scan can be included in differenttraining subsets. As another example, one or more image slices can bepartitioned into different cropped portions, where the different croppedportions are included in different training subsets.

As another example, in an embodiment where the grouping step groups bydifferent types of input data, some or all fields of the medical scanentry can be partitioned and/or pre-processed. For example, image dataof the medical scan can be included in at least one first trainingsubset of the plurality of training subsets. Report data or unstructuredtext data of the medical scan can be included in at least one secondtraining subset of the plurality of training subsets. Raw sensor datacaptured by the modality machine, prior to applying a Fourier togenerate the image data, can be included in at least one third trainingsubset of the plurality of training subsets. Patient history data of themedical scan can be included in at least one fourth training subset ofthe plurality of training subsets.

Performing the grouping step 6005 can include grouping the medical scansbased on grouping criteria received from an administrator via user inputto an interactive interface displayed by a display device of a clientdevice 120. Alternatively or in addition, some or all of the groupingcriteria can be determined automatically, for example, based onautomatically determining features that best distinguish the trainingset of medical scans and grouping based on the set of automaticallydetermined features. For example, statistically significant trends thatdifferentiate medical scans corresponding to various criteria can beutilized to determine optimal groupings, where criteria that mostdifferentiate generation of abnormality data for medical scans aregrouped separately. Alternatively, in response to automaticallydetermining that particular types of medical scans, when groupedtogether to train a single model, negatively impact the accuracy of thesingle model, and that training on the multiple types separatelyimproves the accuracy abnormality data generated instead by separatemodels. As more training data is available overtime and/or as accuracyof different models are evaluated overtime, grouping criteria can berecalibrated to generate new grouping criteria. This can includereassigning groupings automatically to change grouping criteria of atleast one of the groupings based on determining that one or more modelsis too overfit, based on determining new statistically significanttrends that differentiate particular types of scans currently groupedtogether, and/or based on otherwise automatically determining accuracywould be improved by re-designating one or more particular types ofscans into different groupings. Some or all of the trends utilized todetermine groupings can be determined by utilizing the model-assistedannotating system of FIGS. 18A-18E.

Once the training subsets are formed, a training step 6010 can beperformed separately on each of the training subsets 1-K to generate aplurality of sub-models 1-K. Each of the plurality of sub-models is thustrained to process types of medical scans that correspond to thegrouping criteria utilized to determine the corresponding trainingsubset. The training step 6010 can be the same or different for the eachtraining subset. Performing the training step 6010 can includeperforming some or all of training step 5010 of FIG. 15A and/or some orall of training step 1352 of FIG. 7A. Performing the training step 6010can include utilizing the training steps discussed in conjunction withthe model-assisted annotating system of FIGS. 18A-18E. Some or all ofthe plurality of sub-models can correspond to same or different types ofmachine learning models. Some or all of the plurality of sub-models cancorrespond to neural networks with input nodes that correspond to imagedata of the medical scans and/or with output nodes that correspondabnormality data for the medical scans.

Some or all of the plurality of sub-models can be generated to processthe same or different types of input and/or can include the and/ordifferent number of input nodes. For example, the input to some or allof the plurality of sub-models can include pixels or subregions of oneor more image slices of medical scans in the training subset, reportdata or other unstructured text data corresponding to medical scans inthe training subset, raw sensor data captured by the modality machineprior to applying a Fourier to generate image data for medical scans inthe training subset, and/or patient history data for medical scans inthe training subset. One or more of the plurality of sub-models thatutilizes report data and/or unstructured text data as input can utilizea natural language analysis function generated by the medical scannatural language analysis system 114 and/or can train the correspondingones of the plurality of sub-models by training one or more naturallanguage models.

Some or all of the plurality of sub-models can be configured generatethe same or different types of output and/or can include the and/ordifferent number of output nodes. For example, output to some or allinference functions that utilize of the plurality of sub-models can beconfigured to include probabilities indicating whether one or more typesof abnormalities are present, can indicate region of interest datalocalizing one or more detected abnormalities, and/or can includecharacterization data describing size, volume, or other characterizationdata describing one or more detected abnormalities. Some or all of theoutput for some or all of the plurality of sub-models can correspond tosome or all of the leaf nodes of one or more of the prompt decisiontrees of the medical scan hierarchical labeling system 3002 and/or cancorrespond to some or all of the annotation data and/or additionalannotation data generated by utilizing the model-assisted annotatingsystem of FIGS. 18A-18E. Some or all of the output for some or all ofthe plurality of sub-models can correspond to probability matrix data,global probability data, and/or heat map visualization data of themulti-label medical scan analysis system 5002.

In some embodiments, prior to performing the performing the trainingstep 6010, the output labels of a training subset can be pre-processed.For example, in an embodiment where a sub-model is trained to detect aparticular type of abnormality, the output labeling data of thecorresponding training subset can be pre-processed to include a binaryidentifier indicating whether or not the corresponding abnormality typeis present, based on the original output labeling data. Alternatively orin addition, some or all output labels for each medical scan in thetraining subset can be removed if they do not correspond to the intendedoutput. For example, in the embodiment where a sub-model is trained todetect a particular type of abnormality, the output labeling data of thecorresponding training subset can be pre-processed to remove outputlabels that do not correspond to the particular type of abnormality.

Weights, parameters, or other model data that characterize thesub-models 1-K can be stored locally for use by the multi-model medicalscan analysis system 6002 to generate inference data for subsequentlyreceived medical scans, as illustrated in FIGS. 16C-16E. Alternativelyor in addition, some or all of the model data for sub-models 1-K can betransmitted to the medical scan analysis function database 346.

In some embodiments, model parameters for the plurality of sub-models1-K generated by the multi-model medical scan analysis system 6002 aresent to the medical picture archive integration system 2600 for use bythe annotating system 2612. In such embodiments, the central serversystem 2640 can utilize and/or otherwise communicate with themulti-model medical scan analysis system 6002 to generate and/or receivethe model parameters for transmission to the medical picture archivesystem 2600.

FIG. 16B illustrates an embodiment where the plurality of sub-models 1-Kcorrespond to fine-tuned models of a generic model. In particular, thegeneric model can be generated by performing the training step 6015 onthe entire medical scan training set, where the sub-models 1-K aregenerated by performing training steps 6017 on the generic model.Performing training steps 6015 and/or 6017 can include performing someor of training step 6010, and can be the same or different from trainingstep 6010. In particular, performing training step 6017 can includemodifying a plurality of weights of the generic model, for example, bystarting from the generic model and by further training on acorresponding training subset to generate the corresponding fine-tunedmodel. Each fine-tuned model can include the same and/or different typeof output as the generic model. Each fine-tuned model can correspond toan overfitted version of the generic model. Generating some or all ofthe fine-tuned models can include overfitting the generic model tobetter handle a more specific type of input data. Alternatively or inaddition, generating some or all of the fine-tuned models can includeoverfitting the generic model to more accurately generate of output dataand/or to provide output data that includes additional details.

In some embodiments, each fine-tuned model 1-K can correspond to adifferent one of a set of output labels of output labeling data for themedical scans in the training set. In particular, if each output labelin the output labeling data for the generic model corresponds to aprobability that different one of a set of K abnormality types ispresent, each fine-tuned model can correspond to one of the set of Kabnormality types. For example, each fine-tuned model can be configuredto process medical scans to further characterize a type of abnormality,where the output corresponds to characterization labeling data for thetype of abnormality. As another example, each fine-tuned model can beconfigured to process medical scans to more precisely detect a type ofabnormality, where the output corresponds to a probability the type ofabnormality is included in the medical scan. In such embodiments, thegeneric model can correspond to the multi-label model generated by themulti-label analysis system, where each fine-tuned model corresponds toa different one of the labels of the multi-label model.

In some embodiments, generating some or all of the fine-tuned modelsincludes selecting a subset of the training set of medical scans and/orpartitioning data of the training set medical scans. For example, thegrouping step 6005 can be similarly performed as discussed inconjunction with FIG. 16A, and some or all of the fine-tuned models canbe generated by retraining the generic model by utilizing only thecorresponding training subset. Data for medical scans in the medicalscan training set can similarly be partitioned into portions to generatetraining subsets, and the generic model can be retrained on differentportions of data of the medical scans in the training set to generatethe corresponding fine-tuned models.

In some embodiments, one or more of the fine-tuned models is furtherretrained or otherwise fine-tuned to generate a further-fine-tunedmodel. This can be accomplished by again applying the training step 6017to the fine-tuned models, for example, where weights of the fine-tunedmodel are modified. For example, as illustrated in FIG. 16B, trainingstep 6017 is applied to fine-tuned-model 1 to generate eachfurther-fine-tuned model 1-Z. Generating a further-fine-tuned model caninclude by starting from the corresponding fine-tuned model and byfurther training on a smaller training subset and/or different outputlabels to generate the corresponding further-fine-tuned model.Determining the smaller training subset can include selecting onlymedical scans from the training subset that meet further refinedcriteria. Any number of further and further fine-tuned models can begenerated from each fine-tuned model 1-K. For example, the grouping step6005 can be applied to the training subset based on more refinedcriteria to generate a plurality of smaller subsets from the trainingsubset, and each of the plurality of smaller subsets can be utilized toretrain the particular fine-tuned model to generate a plurality offurther-fine-tuned models. In this fashion, a tree of fine-tuned modelscan effectively be generated, where each node in the tree corresponds todifferent training criteria, and where the criteria is more refined tocreate increasingly overfit models deeper and deeper down the tree.

Performing the fine-tuning step to generate one of the plurality offine-tuned models corresponding to the first one of the plurality ofabnormality types can include utilizing additional or otherwisedifferent output labeling data of a subset of the plurality of medicalscans. For example, each of the subset of the plurality of medical scanscan have corresponding labeling data indicating the first one of theplurality of abnormality types is present in the each of the subset ofthe plurality of medical scans, as well as additional labeling dataindicating characterization labels characterizing the first one of theplurality of abnormality types in the each of the subset of theplurality of medical scans. These characterization labels can beutilized as output data to generate the fine-tuned model. Each of thecharacterization labels can indicate whether or not a correspondingcharacterization category is present and/or can indicate a value or oneof a discrete set of options for the corresponding characterizationcategory. Some or all of the characterization labels can indicateabnormality classification data 445 of a corresponding abnormalityclassification category.

In some embodiments, each of a plurality of further-fine-tuned modelscorrespond to internal nodes of one or more prompt decision trees. Inparticular, a subset of medical scans utilized to generate one of thefine-tuned models 1-K from the generic model can correspond to medicalscans with output labeling data that compares favorably to a firstinternal node of a prompt decision tree. This fine-tuned model can befurther refined to generate a further-fine-tuned model by retraining ona more refined subset, corresponding to medical scans with outputlabeling data that compares favorably to a second internal leaf nodebranching from the first internal node root node of a prompt decisiontree. This further-fine-tuned model can be further and furtherfine-tuned by retraining on further refined models by further filteringthe training set by only including medical scans that compare favorablyto corresponding internal nodes deeper down the tree towards the leafnode. Some or all further-fine-tuned models can be generated byretraining subsequently generated further-fine-tuned models on furtherrefined subsets corresponding to medical scans with output labeling datathat compares favorably to a leaf node leaf node branching from a finalinternal node root node of a prompt decision tree.

In some embodiments, generating one or more of the fine-tuned modelsand/or further-fine-tuned models includes requesting additional trainingdata. For example, additional training data can be requested in responseto determining the subset of training data utilized to retrain thegeneric model or another previous model is too small, does not include avast enough representation of medical scans, and/or when additionalmedical scans are otherwise determined to be necessary for training. Themulti-label medical scan analysis system can generate a transmission tothe medical scan database 342 for additional medical scans that meetcriteria corresponding to a fine-tuned model and/or further-fine-tunedmodel identified to need more training data. Alternatively, themulti-label medical scan analysis system can forego generating theidentified fine-tuned model and/or further-fine-tuned model until a timethat the necessary additional training data is received via the networkand/or is otherwise determined to be available. Alternatively, themulti-label medical scan analysis system can generate the identifiedfine-tuned model and/or further-fine-tuned model utilizing thecorresponding training data that is available, and can generate anupdated version of the identified fine-tuned model and/orfurther-fine-tuned model once a sufficient amount of additional trainingdata is received.

FIGS. 16C-16E illustrate embodiments of utilizing the multi-modelmedical scan analysis system 6002 to generate inference data on newmedical scans. As illustrated in FIG. 16C, a new medical scan can bereceived for processing, for example, from the medical scan database342. A sub-model selection step 6020 can be performed to determine whichsub-models 1-J will be applied to the medical scan, where J is less thanor equal to K and where J is greater than or equal to one. For example,the header of the medical scan can be processed to determine a modality,anatomical region, or other scan classifier data 420 of the medicalscan, and one or more sub-models correspond to the determined modality,anatomical region, or other scan classifier data 420 is selected. Asanother example, input quality assurance function 1106 can be utilizedto determine a scan category 1120 for the new medical scan, and one ormore sub-models can be selected based on the scan category 1120. Asub-model can be determined to correspond to a medical scan thedetermined modality, anatomical region, scan category 1120, or otherscan classifier data 420 when medical scans from the training set weregrouped into the training subset for the sub-model based on the sameand/or substantially similar criteria. For example, one or moresub-models trained on a training subset corresponding to chest x-rays ina particular hospital setting can be selected in the sub-model selectionstep 6020 in response to determining the incoming medical scan is achest x-ray captured in the same particular hospital setting. In someembodiments, all of the sub-models 1-K are applied to some or allincoming medical scans.

Once the sub-models 1-J are selected, a set of corresponding inferencefunctions 1-J can be performed on the medical scan to generatecorresponding abnormality data 1-J. Some or all of the inferencefunctions 1-J can utilize medical scan analysis functions and/or otherinference functions discussed herein. Some or all of the inferencefunctions can utilize the inference function 5020 of the multi-labelmedical scan analysis system.

Each of the abnormality data 1-J can indicate at least one abnormalitydetected in the medical scan and/or can indicate the medical scan isnormal. Some or all of the abnormality data 1-J can indicate its ownprobability matrix data, global probability data, and/or heat mapvisualization data discussed in conjunction with the multi-label medicalscan analysis system 5002. Some or all of the abnormality data 1-J canindicate probabilities that one or more types of abnormalities arepresent and/or can further characterize and/or localize one or moreparticular types of abnormalities.

A final inference function can utilize the set of abnormality data 1-Jto generate final abnormality data. The final abnormality data caninclude some or all of the abnormality data 1-J. For example, the finalabnormality data can indicate a plurality of different abnormalitiesindicated in different ones of the abnormality data 1-J. Alternatively,the final abnormality data can include a final determination for atleast one abnormality based on consolidating all of the abnormality data1-J. This can include determining a medical scan is normal in responseto a threshold number of the abnormality data 1-J indicating the medicalscan is normal. This can include determining a medical scan includes anabnormality in response to a threshold number of the abnormality data1-J indicates an abnormality is present. This can include determining amedical scan includes an abnormality in response to one of theabnormality data 1-J indicating an abnormality is present with aprobability that exceeds a threshold. This can include determining thatthe medical scan includes an abnormality with a high probability valueas a result of multiple ones of the abnormality data 1-J indicating anabnormality is present with lower probabilities than the highprobability value. This can include computing a probability that anabnormality is present as a weighted average of probabilities that theabnormality is present in abnormality data 1-J. Ones of the abnormalitydata 1-J can be assigned higher weights in response to determining thecorresponding sub-models are more accurate than other sub-models, whereother ones of the abnormality data 1-J generated by utilizing the other,less accurate, sub-models are assigned lower weights. Weighted averagescan be similarly applied to other features of the output labeling data,for example, to generate final probabilities that a detected abnormalitycorresponds to a particular type of abnormality, to generate finalprobability that one or more characterizing features of a detectedabnormality are present, etc.

A first threshold required to determine that an abnormality is presentin the final abnormality data can be equal to a first probability valuewhen only one of the abnormality data 1-J indicates an abnormality ispresent. A second threshold probability required to determine whether anabnormality is present can be equal to a second probability value whenmore than one of the abnormality data 1-J compare favorably to thesecond threshold probability, where the second probability value islower than the first probability value. Subsequent thresholdprobabilities required to determine whether an abnormality is presentcan be equal to subsequently lower values as more and more ones of theabnormality data 1-J indicate the abnormality is present. For example,if abnormality data generated by utilizing five inference functionsindicates probabilities that an abnormality is present withprobabilities 10%, 13%, 81%, 90%, and 2%, an abnormality can bedetermined to be present when a second threshold probability is equal to80% because two of the five abnormality data indicate probabilitiesgreater than 80%. As another example, if abnormality data generated byutilizing five inference functions indicates probabilities that anabnormality is present with probabilities 10%, 71%, 71%, 72%, and 2%, anabnormality can be determined to be present when a third thresholdprobability is equal to 70% because three of the five abnormality dataindicate probabilities greater than 70%, even though the secondprobability threshold requiring two of the five abnormality dataindicate probabilities greater than 80% is not met. This scheme can besimilarly applied to other features of the output labeling data, forexample, to determine whether detected abnormality corresponds to aparticular type of abnormality, to determine whether one or morecharacterizing features of a detected abnormality are present, etc.

In some embodiments, each of the sub-models 1-K are trained to detectdifferent types of abnormalities. For example, consider the embodimentwhere each sub-model 1-K is a fine-tuned model 1-K or afurther-fine-tuned model generated as discussed in conjunction with FIG.16B. Each inference function 1-J can be applied to the same input data,and each of the output labeling data 1-J can indicate a probability thatone of the abnormalities types is present. The final abnormality datacan indicate a global probability indicating a probability that anyabnormality is present, and/or can indicate a global binary identifierindicating whether or not an abnormality is present. For example, theglobal binary identifier can indicate an abnormality is present inresponse to one of the 1-J output labeling data indicating acorresponding type of abnormality is present with a probability thatexceeds a threshold, where the threshold is the same or different foreach of the 1-J abnormality types. As another example, the globalprobability can be computed as a weighted average of the probabilitiesthat each of the types of abnormalities is present. The globalprobability can utilize the global abnormality probability discussed inconjunction with the medical scan triaging system 8002.

In some embodiments, data for new medical scans can be partitioned intodifferent portions, and the same or different set of sub-models 1-J canbe selected for each of the different portions and can be applied toeach different portion to generate the abnormality data. The differentportions can correspond to different views of the same or differentanatomical region. Alternatively or in addition, the different portionscan correspond to different types of data corresponding to the medicalscan. For example, each of the sub-models 1-J can process a differentsequence of an incoming MM scan and/or can process different views of aset of x-rays. As another example, one sub-model can process reportdata, and at least one other sub-model can process the correspondingimage data, and/or at least one other sub-model can process raw sensordata. As another example, each of the sub-models 1-J can processdifferent subsets of the image slices of the image data, and/or each ofthe sub-models 1-J can process different regions of image data for oneor more the image slices. The final abnormality data can be generated byconsolidating abnormality data generated for different types of inputand/or for different portions of the image data to determine whether anabnormality is present, to determine one or more types of abnormalitythat are present, and/or to characterize one or more types ofabnormalities determined to be present.

As another example, an entire study for a patient can be received thatincludes a plurality of different medical scans and/or reports for thepatient. This can include longitudinal data collected for the patientover time and/or can include different modalities of medical scansand/or medical scans captured for different anatomical regions. Thestudy can be partitioned into multiple medical scans and/or reports, andthe same or different sub-models 1-J can be applied to each medical scanand/or report. All of the abnormality data generated by applyingsub-models to all of the medical scans and/or reports in the study canbe similarly consolidated in generating the final abnormality data.

The probability of one abnormality pattern identified by utilizing onemodel can influence probability of other abnormality patterns and/or caninfluence probability of characterization of other abnormalities ingenerating the final abnormality data. The probability of an abnormalitybased on one sequence and/or modality can influence probability of thesame abnormality, or additional types of abnormalities corresponding todiffering pathologies, detected based on additional sequences and/ormodalities. This can be utilized to consolidate the set of abnormalitydata to generate the final abnormality data. Generating the finalabnormality data can include utilizing a Bayesian model to generate afinal probability that an abnormality is present, given the plurality ofprobabilities of the set of abnormality data for different types and/orportions of input such as different views, sequences, anatomicalregions, report data, or other data for the medical scan. Alternativelyor in addition, generating the final abnormality data can includeutilizing a Bayesian model to generate a final probability that each ofa plurality of abnormality types are present given a plurality ofprobabilities that each of a plurality of abnormality types are present.

Alternatively or in addition, generating the final abnormality data caninclude utilizing a plurality of known correlations between differenttypes of abnormalities, where the final abnormality data is generatedbased on a known correlation value between the first type of abnormalityand the second type of abnormality. For example, if the final inferencefunction utilizes a model trained by the multi-model medical scananalysis system 6002, the plurality of known correlations can beautomatically determined as a result of generating training model.

Consider an example where a first one of the subset of the set ofsub-models is trained to detect a first type of abnormality and a secondone of the subset of the set of sub-models is trained to detect a secondtype of abnormality. A first one of the set of abnormality data isgenerated as output of a first one of the subset of the set of inferencefunctions that corresponds to the first one of the subset of the set ofsub-models. A second one of the set of abnormality data is generated asoutput of a second one of the subset of the set of inference functionsthat corresponds to the first one of the subset of the set ofsub-models. Suppose the first one of the set of abnormality dataindicates a probability that the first type of abnormality is present,and the second one of the set of abnormality data indicates a secondprobability that second type of abnormality is present. The finalinference function can utilize a Bayesian model and/or a knowncorrelation between the first and second type of abnormality todetermine final probabilities that the first and second types ofabnormalities are present. For example, the final abnormality data canindicate an increase in the probability that the second type ofabnormality is present in response to the set of abnormality dataindicating the probability that the first type of abnormality is presentcompares favorably to a detection probability threshold and in responseto the known correlation between the first type of abnormality and thesecond type of abnormality comparing favorably to a correlationthreshold. As another example, the final abnormality data can indicate adecrease in the probability that the second type of abnormality ispresent in response to the set of abnormality data indicating theprobability that the first type of abnormality is present comparesunfavorably to a detection probability threshold and in response to theknown correlation between the first type of abnormality and the secondtype of abnormality comparing favorably to a correlation threshold.

As illustrated in FIG. 16D, some or all of the sub-models can be appliedin sequence, and can utilize abnormality data generated by previouslyapplied sub-models. For example, a first one of the 1-J inferencefunctions can be applied to the medical scan to generate abnormalitydata 1. A second one of the 1-J inference functions can be applied tothe same or different portion of data the medical scan and/or to some orall of the abnormality data 1 to generate abnormality data 2. Thisprocess can continue until final abnormality data is generated, onceinference function J is applied. The selection and/or ordering of the Jinference function can be determined in the sub-model selection step6020 and can be fixed until the final abnormality data is generated.Alternatively, the next inference function to be applied can bedetermined as output of applying a selection step after each abnormalitydata 1, 2, . . . , J−1 is generated, as a function of the abnormalitydata generated thus far.

In some embodiments, a different portion of the medical scans areapplied to each inference function. In particular, smaller sub-regionscan be selected from previous sub-regions of the image data for eachsubsequently applied inference function, based on localization dataindicated in each subsequently generated abnormality data that furtherlocalizes a detected abnormality. For example, generating a sub-regionfrom a previous sub-region can include selecting a proper subset ofimage slices and/or can include selecting a cropped portion of imagedata in one or more slices based on localization in the abnormality dataof one or more previously applied sub-models. Some or all sub-regionscan be successively smaller in size based on localization data in theabnormality data generated thus far, and/or can include different imagedata not included in a previous sub-region based on localization data inthe abnormality data generated thus far.

FIG. 16E illustrates a particular example of sequentially applyinginference functions. A first inference function, such as genericinference function 6025 corresponding to the generic model generated asdiscussed in conjunction with FIG. 16B, can be applied to the medicalscan to generate initial abnormality data. Alternatively, the firstinference function can be selected based on scan classier data of themedical scan by applying the sub-model selection step 6020. The initialabnormality data can indicate one of a plurality of types ofabnormalities, and inference function X can be selected based ondetermining a corresponding one of the sub-models 1-K that is trained todetect, localize, and/or further characterize the detected type ofabnormality in the initial abnormality data. For example, as illustratedin FIG. 16E, sub-model selection step 6020 can be applied to the initialabnormality data to select one of the fine-tuned models generated asdiscussed in conjunction with FIG. 16B, where the inference function Xcorresponds to the selected fine-tuned model. The additional abnormalityoutput can indicate that the type of abnormality is detected with ahigher confidence, or can alternatively indicate that the detectedabnormality does not likely correspond to the type of abnormalityinitially detected, and perhaps corresponds to a different type. Theadditional abnormality output can localize and/or further localize thetype of abnormality in the image data. The additional abnormality outputcan include characterization data for the type of abnormality. Inparticular, the characterization data can be specific to the type ofabnormality, where different ones of the fine-tuned models generatedifferent types of characterization data specific to their correspondingtype of abnormality.

The additional abnormality data can indicate a plurality of probabilityvalues indicating whether each of a plurality of characterization typescharacterize one of the plurality of abnormality types indicated in theinitial abnormality data, as a result of applying an inference functioncorresponding to the one of the plurality of abnormality types. Theadditional abnormality data can indicate a subset of the plurality ofcharacterization types determined to characterize the first one of theplurality of abnormality types in response to corresponding ones of theplurality of probability values comparing favorably to acharacterization threshold. The plurality of characterization types cancorrespond to a plurality of characterization labels included in outputlabel data utilized to train the inference function.

While not illustrated in FIG. 16E, one or more further-fine-tuned modelscan be iteratively selected based the most recently generated additionalabnormality data. The selection of further-fine-tuned models cancorrespond to models trained to further detect, characterize, and/orlocalize abnormalities. This can be determined based on a selecting aninternal node and/or leaf node branching from a previous internal node,where the selected internal node and/or leaf node closely matches theadditional abnormality data generated thus far, where thefurther-fine-tuned model was trained by a training subset that comparesfavorably to the labeling data that corresponds to findings in theadditional abnormality data. Subsequently selected further-fine-tunedmodels can continue to be determined, based continuing to propagate downa prompt decision tree, where each next node is determined from the setof nodes branching from a current node based on determining which optionmost closely corresponds to the most recently generated additionalabnormality data. This process can continue until a leaf node isreached, until no more models are available, and/or until probabilitiesof the additional abnormality data compare unfavorably to a threshold,indicating that further characterization is likely to be inconclusiveand/or indicating that the abnormality cannot be further characterized.

If multiple abnormalities are detected in the one or more medical scansand/or reports, these multiple abnormalities can be processedseparately. This can include consolidating the multiple abnormalitiesseparately in the final abnormality data when applying the finalinference function. Alternatively or in addition, this can includeselecting separate fine-tuned functions for each abnormality detected byutilizing the generic algorithm, allowing each abnormality to beseparately confirmed, characterized, and/or localized further.

In various embodiments, the multi-model medical scan analysis system6002 includes at least one processor and a memory that storesoperational instructions that, when executed by the at least oneprocessor, cause the multi-model medical scan analysis system to performthe operations discussed herein. In particular, the multi-model medicalscan analysis system can be operable to receive, via a receiver, aplurality of medical scans. A plurality of training sets can begenerated from the plurality of medical scans. Each of a set ofsub-models can be generated by performing a training step on acorresponding one of the plurality of training sets. A new medical scancan be received via the receiver, and subset of the set of sub-modelscan be selected based on the new medical scan. A set of abnormality datacan be generated by applying a subset of a set of inference functions onthe new medical scan, where the subset of the set of inference functionsutilize the subset of the set of sub-models. Each of the set ofabnormality data can be generated as output of performing one of thesubset of the set of inference functions on the new medical scan. Finalabnormality data can be generated by performing a final inferencefunction on the set of abnormality data. The final abnormality data canbe to a client device for display via a display device.

In various embodiments, the multi-model medical scan analysis system isoperable to receive, via a receiver, a plurality of medical scans. Eachof the plurality of medical scans can include corresponding labelingdata indicating whether each of a plurality of abnormality types arepresent in the each of the plurality of medical scans. A generic modelcan be generated by performing a training step on image data of theplurality of medical scans and the corresponding labeling data. Aplurality of fine-tuned models can be generated, where each of theplurality of fine-tuned models is generated by performing a fine-tuningstep on the generic model, and where of the plurality of fine-tunedmodels corresponds to one of the plurality of abnormality types. A newmedical scan can be received, via the receiver. Abnormality detectiondata can be generated for the new medical scan by performing a genericinference function on image data of the new medical scan. The genericinference function can utilize the generic model, and the abnormalitydetection data can indicate a plurality of probability values that eachindicate a probability that a corresponding one of the plurality ofabnormality types is present in the new medical scan. A first one of theplurality of abnormality types can be determined to be detected in thenew medical scan by determining a corresponding one of the plurality ofprobability values compares favorably to a detection threshold. One ofthe plurality of fine-tuned models that corresponds to the first one ofthe plurality of abnormality types can be selected. Additionalabnormality data can be generated for the new medical scan by performinga fine-tuned inference function on the image data of the new medicalscan, where the fine-tuned inference function utilizes the one of theplurality of fine-tuned models. The additional abnormality data can betransmitted to a client device for display via a display device.

The multi-model medical scan analysis system 6002 can be implemented asa location-based medical scan analysis system 6004, as shown in FIG. 17.Factors such as patient population, imaging techniques/machinery,specialist training, definition of normal and abnormal, and/or otherfactors can vary across different geographic regions. These distinctionscan be as small as between different hospitals, or as large as betweendifferent countries. Similarly, the type of hospital setting can affecttrends of image data. In particular, whether the medical scan was takenin an inpatient setting, outpatient setting, emergency care setting, orother setting can further vary imaging data, for example, where testtubes are present in some settings and not others.

The location-based medical scan analysis system 6004 improves existingsystems by presenting a solution to this problem. In particular, thelocation-based medical scan analysis system 6004 can be utilized tooverfit one or more models based on specific trends of a particularlocation. The model can be overfit by hospital, by a larger geographicregion, and/or by hospital setting to generate a plurality of models.Generating an overfit model for a particular location can be beneficialif the model is always run in the particular location and/or is alwaysperformed on medical scans generated in the particular location.

The multi-model medical scan analysis system 6002 can be utilized togenerate such location-based models. In particular, some or all of thesub-models 1-K can correspond to different ones of a plurality ofparticular locations. As used herein, a particular location cancorrespond to any geographic region, any hospital-based location, and/orany hospital setting. The grouping step 6005 can be applied to trainingdata to group medical scans based on their originating location tocreate training subsets grouped by geographic region, by hospital-basedlocation, and/or by hospital setting. Some or all of the groupings caninclude one or more of the same medical scans. Alternatively, all of thegroupings can be mutually exclusive. The grouping step 6005 can groupmedical scans only by particular location, and/or can also utilize othergrouping criteria discussed in conjunction with FIG. 16A.

A training subset grouped by hospital setting can include medical scansfrom a plurality of different hospitals and/or geographic regions, solong as they were performed in the corresponding hospital setting. Forexample, an inpatient training subset can include medical scans taken inan inpatient setting across many different hospitals and/or differentgeographic regions. Alternatively, a training subset grouped by hospitalsetting can include medical scans from the same hospital and/or samegeographic region. For example, multiple inpatient training subsets canbe generated, each including medical scans taken in an inpatient settingin same hospital and/or in the same geographic region.

Determining which particular location each medical scan in the medicalscan training set originated from can be accomplished by determining alocation associated with an entity from which the medical scan wastransmitted from. For example, if medical scans are transmitted to thelocation-based medical scan analysis system 6004 from differentdatabases and/or different local systems 6050, a location associatedwith the database and/or local system from which each medical scan wastransmitted can be utilized to determine the particular locationassociated with the medical scan. Alternatively or in addition, headerdata or other metadata associated with the medical scan, such asoriginating entity data 423 and/or geographic region data 424 can beutilized to determine the particular location. While not shown in FIG.4A, medical scan entries can further indicate which of a plurality ofhospital settings the medical scan was captured. Alternatively or inaddition, patient data indicating a geographic region from which thepatient is from can be utilized to determine the geographic regionmedical scan for the patient should be designated.

Training step 6010 can similarly be applied to all of the location-basedtraining subsets to generate corresponding location-based sub-models1-K. In some embodiments, a generic model is first created utilizingsome or all of the medical scan training data, and each of thelocation-based sub-models correspond to fine-tuned models generated byretraining the generic model on the corresponding location-basedtraining subset. In some embodiments, a location-based hierarchical treeis utilized to further refine one or more of the fine-tuned models. Inparticular, the generic model can utilize all of the training set ofmedical scans, a first sublayer of models can correspond to fine-tunemodels generated by retraining the generic model on training subsetscorresponding to large geographic regions. A second sublayer of modelscan correspond to further-fine-tuned models corresponding to smallergeographic regions within the large geographic regions, where each ofthe second sublayer of models is generated by retraining one of thefine-tuned models on a training subset corresponding to a smallergeographic region within the large geographic region. For example, thefirst sublayer can correspond to models trained by country, and thesecond sublayer can correspond to models trained by hospitals withineach country. Any number of sublayers of further-fine-tuned models bylocation can extend from the generic model.

The location-based medical scan analysis system 6004 can automaticallydetermine grouping criteria for some or all of the location-basedsub-models. In particular, the location-based medical scan analysissystem 6004 can automatically determine to create distinct models acrossdifferent geographic locations in response to determining statisticallysignificant discrepancies between the locations. For example, thestatistically significant discrepancies can be automatically determinedin response to generating a generic model and by evaluating theperformance across different particular locations. Any statisticallysignificant discrepancies can be utilized to automatically groupcorresponding locations separately to create the correspondinglocation-based sub-models. In some embodiments, once a plurality offine-tuned models are generated, some or all of the of fine-tuned modelscan be automatically evaluated for location-based discrepancies toautomatically determine grouping criteria to generate one or morefurther-fine-tuned models. As more training data is available overtimeand/or as accuracy of different models are evaluated overtime,sub-models can be retrained. This can include reassigning groupingsautomatically to change grouping criteria of at least one of thegroupings based on determining that one or more models is too overfit,based on determining new statistically significant trends thatdifferentiate particular locations currently grouped together, and/orbased on otherwise automatically determining accuracy would be improvedby re-designating one or more particular locations into differentgroupings.

Alternatively or in addition, some or all of the grouping criteria canbe generated in response to receiving one or more requests for alocation-based model from client devices associated with one or moreparticular locations. For example, a client device associated with ahospital or other medical entity, such as a client device associatedwith a local system 6050, can request its own location-based model foruse on medical scans originating from that hospital or other medicalentity. Alternatively or in addition, some or all of the groupingcriteria can be generated based on administrator input to an interactiveinterface displayed via a display device of a client device 120communicating with the location-based medical scan analysis system 6004.

In some embodiments, as illustrated in FIG. 17, a plurality of localsystems 6050 communicate with the location-based medical scan analysissystem 6004. The plurality of local systems can each be located in, orcan otherwise be associated with, one of a plurality of particularlocations. Each of the plurality of sub-models 1-K can be transmitted toa corresponding one of the local systems 6050, for example, as weight,parameters, inference function execution data, or other datacharacterizing the corresponding sub-models. The local system canreceive its designated sub-model from the location-based medical scananalysis system 6004 via a network interface 6055. The local system canperform the corresponding inference function by its own processingsystem 6052, which can include one or more processors and memory thatstores its own executable instructions that, when executed by the one ormore processors, causes the processing system 6052 to utilize thesub-model to perform the corresponding inference function on new medicalscans to generate corresponding abnormality data. For example, theprocessing system 6052 can be implemented by utilizing one or moresubsystems 101. The processing system 6052 can communicate with one ormore client devices 120, for example, via a wired and/or wirelessnetwork of the local system. For example, the abnormality data generatedfor new medical scans can be displayed to a user via an interactiveinterface displayed via a display device of the client device, forexample, by utilizing the medical scan assisted review system 102.

At least one of the local processing systems can include its own medicalscan database 342, for example, populated by medical scans generated inthe particular location. For example, one or more modality machineslocated in the particular location can send imaging data to the medicalscan database 342. The abnormality data generated for new medical scansby utilizing the sub-model received from the location-based medical scananalysis system 6004 can be sent to the medical scan database forstorage in conjunction with the new medical scans. Alternatively, theabnormality data can be transmitted via the network interface 6055 toother databases and/or systems communicating via network 150. Forexample, the new medical scans and/or abnormality data can betransmitted back to the location-based medical scan analysis system6004, and the location-based medical scan analysis system 6004 canutilize the new medical scans and/or abnormality data to retrain one ormore sub-models and/or to retrain the generic model.

In some embodiments, each training subset 1-K utilized to train eachsub-model is received separately from each of the of the localprocessing systems 6050 as its own medical scan training set 1-K. Forexample, each medical scan training set 1-K can be retrieved by a localsystem 6050 from the own medical scan database 342 for transmission tothe location-based medical scan analysis system via the networkinterface 6055.

In some embodiments, multiple location-based sub-models can be trainedby the location-based medical scan analysis system 6004 for each localprocessing system. For example, the location-based medical scan analysissystem 6004 can receive medical scan training set 1 from a first one ofthe local systems, can perform grouping step 6005 as described inconjunction with FIG. 16A to generate a plurality of training subsetsfrom the medical scan training set 1 to generate a plurality ofsub-models particular to the first one of the local systems, where theplurality of sub-models are each configured to process different typesof medical scans and/or generate different types of inference data asdiscussed herein. Alternatively, some or all of a plurality oflocation-independent sub-models generated to process different types ofmedical scans and/or generate different types of inference data can befurther refined by location by utilizing medical scan training set 1 togenerate the location-based sub-models. The set of sub-models overfittedfor the particular location corresponding to the first local system 6050by utilizing medical scan training set 1 can be sent back to the firstlocal system 6050. The first local system 6050 can perform the sub-modelselection step 6020 itself in response to determining to process a newmedical scan to determine which of the set of its set of sub-models toutilize. The set of sub-models can be stored by the first local system6050 in its own medical scan analysis function database 346. Each localsystem can receive the same or different set of sub-models overfittedfor their own respective locations based on their respective trainingsets, and can similarly store their set of sub-models in their ownmedical scan analysis function database 346 or other memory.

Some or all of the plurality of local systems 6050 can be associatedwith particular hospitals or medical entities. Some or all of theplurality of local systems 6050 can be implemented by utilizing one ormore subsystems 101 and/or by utilizing one or more client devices 120.Some or all of the of the plurality of local systems 6050 can beimplemented by utilizing a medical picture archive integration system2600, for example, where the central server system 2640 is implementedby utilizing the location-based medical scan analysis system 6004 and/orotherwise communicates with the location-based medical scan analysissystem 6004. For example, the annotating system 2612 can receive, store,and utilize one or more sub-models overfitted for the locationassociated with the corresponding medical picture archive integrationsystem 2600, received from the location-based medical scan analysissystem 6004 and stored as the inference functions 1-K stored in memory2684. The medical scan training set utilized for a local system can beanonymized by its de-identification system 2608 for transmission to thelocation-based medical scan analysis system 6004.

In various embodiments, a location-based medical scan analysis system6004 includes at least one processor and a memory that storesoperational instructions that, when executed by the at least oneprocessor, cause the location-based medical scan analysis system toreceive, via a receiver, a plurality of medical scans. A generic modelis generated by performing a training step on image data of theplurality of medical scans. A plurality of location-based subsets of theplurality of medical scans is generated. Each of the plurality oflocation-based subsets is generated by including ones of the pluralityof medical scans with originating locations that compare favorably tolocation grouping criteria for the each of the plurality oflocation-based subsets. A plurality of location-based models isgenerated. Each of the plurality of location-based models is generatedby performing a fine-tuning step on the generic model, utilizing acorresponding one of the plurality of location-based subsets. Inferencedata is generated for a new medical scan by performing an inferencefunction that utilizes one of the plurality of location-based models onthe new medical scan. An originating location associated with the newmedical scan compares favorably to the location grouping criteria forone of the plurality of location-based subsets utilized to generate theone of the plurality of location-based models. The inference data istransmitted to a client device for display via a display device.

FIGS. 18A-18E illustrate an embodiment of a model-assisted annotatingsystem 7002. One or more models can be continually updated over time asadditional training data becomes available, by utilizing annotationsgenerated via user input to a client device. Some or all of thesehuman-generated annotations for subsequent sets of training data can beassisted via display of, and/or can be supplemented by,automatically-generated annotations generated by utilizing a currentversion of one or more models. The human-generated annotations canconfirm accuracy of these automatically-generated annotations, canprovide corrections and/or edits to the automatically-generatedannotations, and/or utilize the automatically-generated annotations toprovide additional, more detailed information. These human-generatedannotations for subsequent sets of training data can then be utilized toupdate one or more models, and this process can be repeated over time tofurther improve models.

As shown in FIGS. 18A-18E, the model-assisted annotating system 7002 cancommunicate bi-directionally, via network 150, with the medical scandatabase 342, medical scan analysis function database 346, with otherdatabases of the database storage system 140, with one or more clientdevices 120, and/or, while not shown in FIGS. 18A-18E, with one or moresubsystems 101 of FIG. 1, with the central server system 2640, and/orwith the medical picture archive system 2600.

In some embodiments, the model-assisted annotating system 7002 is anadditional subsystem 101 of the medical scan processing system 100,implemented by utilizing the subsystem memory device 245, subsystemprocessing device 235, and/or subsystem network interface 265 of FIG.2A. For example, the model-assisted annotating system 7002 can beimplemented by utilizing the medical scan image analysis system 112, themulti-label medical scan analysis system 5002, and/or the multi-modelmedical scan analysis system 6002 to train one or more computer visionmodels and/or to perform one or more inference functions by utilizingthe one or more computer vision models. Furthermore, the generation ofhuman-generated annotation data via user input to interactive interfacesby utilizing client devices 120, as illustrated in FIGS. 18A and 18C,can be implemented by utilizing the medical scan assisted review system102, the medical scan annotating system 106, and/or the medical scanhierarchical labeling system 3002.

In some embodiments, the model-assisted annotating system 7002 utilizes,or otherwise communicates with, the central server system 2640. Forexample, the medical scan database 342 can be populated withde-identified data generated by the medical picture archive integrationsystem 2600. The model-assisted annotating system 7002 can receivede-identified medical scans of the training set with their correspondingannotation data, diagnosis data, and/or medical reports directly fromthe medical picture archive integration system 2600, for example, wherethe annotation data, diagnosis data, and/or medical reports are utilizedto determine the medical labels for medical scans in the training set.As another example, the model-assisted annotating system 7002 canperform one or more inference functions on de-identified medical scansreceived from the medical picture archive integration system 2600, andabnormality data or other inference data generated as output of theplurality of the one or more inference functions can be assigned to themedical scan in the medical picture archive integration system 2600. Asanother example, the model-assisted annotating system 7002 can requestde-identified medical scans, annotation data, and/or reports that matchrequested criteria for the training set and/or for new medical scans tobe labeled. In some embodiments, some or all of the model-assistedannotating system 7002 is implemented by utilizing other subsystems 101and/or is operable to perform functions or other operations described inconjunction with one or more other subsystems 101.

As illustrated in FIG. 18A, a first set of medical scans can beretrieved from a medical scan database for transmission to a pluralityof client devices 120. The model-assisted annotating system 7002 canautomatically determine which client devices the first set of medicalscans are transmitted to and/or which medical scans in the first set ofmedical scans are sent to each client device, for example, based onperformance data and/or qualification data of corresponding user and/orbased on means utilized by the medical scan annotating system 106.

Each of the client devices can display one or more medical scans of thefirst set to a user via an interactive interface 7075, and can generateannotation data based on user input to the interactive interface inresponse to a prompt displayed by the interactive interface. Theinteractive interface 7075 can utilize features of the medical scanassisted review system 102 and/or the medical scan hierarchical labelingsystem 3002, for example where the interactive interface 7075 utilizessome or all features of interactive interface 3075 discussed inconjunction with the medical scan hierarchical labeling system 3002and/or some or all features discussed in conjunction with FIGS. 13C-13V.For example, the medical scan 1.1 in the first medical scan set isdisplayed via a first client device, and corresponding annotation data1.1 is generated by the first client device by utilizing user input tothe interactive interface. Each annotation data 1.1-1.N can correspondto structured, standardized labeling data, such as the labelingdiscussed in conjunction with the medical scan hierarchical labelingsystem 3002. The annotation data 1.1-1.N can include any structured orunstructured data, and can indicate any of the diagnosis data 440discussed herein. A first set of annotation data, which includes theannotation data 1.1-1.N corresponding to medical scans 1-N of the firstset of medical scans, can be transmitted to the medical scan databasefor storage in conjunction with the first set of medical scans and canbe transmitted to the model-assisted annotating system 7002.

As illustrated in FIG. 18B, the model-assisted annotating system 7002can utilize this first set of medical scans and corresponding annotationdata to train a model, which can be utilized in performing any of themedical scan analysis functions or other inference functions discussedherein. For example, the model can correspond to a computer vision modelthat utilizes a neural network, where input labels correspond to imagedata of the medical scans and where output labels correspond to some orall of the annotation data. The training step 7010 utilized to train themodel can be implemented by utilizing some or all of training step 5010of the multi-label medical scan analysis system, for example, where themodel corresponds to a multi-label model utilized to perform inferencefunction 5020. Alternatively or in addition, the training step 7010utilized to train the model can be implemented by utilizing some or allof training step 6010, for example, where a plurality of sub-models 1-Kare trained and are utilized to perform inference functions 1-K.Alternatively or in addition, the training step 7010 utilized to trainthe model can be implemented by utilizing some or all of training step1352 of FIG. 7A. Model parameters for the trained model can be storedlocally for further use by the model-assisted annotating system 7002and/or can be transmitted to the medical scan function database for useby other subsystems 101 as any medical scan analysis function and/orinference function as discussed herein.

Over time, this model can be implemented by the model-assistedannotating system 7002 and/or other subsystems 101 to generateannotation data for new medical scans. As illustrated in FIG. 18C, asecond set of medical scans can be received from the medical scandatabase, for example, where the second set of medical scans includesnew medical scans not included in the first set of medical scans, suchas more recent medical scans that were not previously available orotherwise were determined to not be included in the first set of medicalscans. An inference function 7020 can be performed on each of the secondset of medical scans to automatically generate annotation data for eachof the second set of medical scans. Inference function 7020 can utilizethe model generated by performing training step 7010 in FIG. 17D.Inference function 7020 can utilize inference step 6030 to perform oneor more inference functions 1-J selected for the medical scan and/or afinal inference function, where the annotation data corresponds to thefinal abnormality data. Inference function 7020 can utilize inferencefunction 5020, global labeling function 5030, and/or heat map generatorfunction 5040, where the annotation data includes the probability matrixdata, global probability data, and/or heat map visualization datadiscussed in conjunction with FIGS. 14B and 14C. Inference function 7020can utilize inference step 1354, detection step 1372, abnormalityclassification step 1374, similar scan identification step 1376, and/ordisplay parameter step 1378 as discussed in conjunction with FIG. 7B.The annotation data generated as output to the inference function caninclude probabilities and/or can correspond to structured labeling dataas discussed in conjunction with the medical scan hierarchical labelingsystem, for example, where the annotation data indicates probabilitiesfor one or more internal nodes and/or leaf nodes of a prompt decisiontree to characterize one or more abnormalities. Each of the second setof annotation data can indicate any of the diagnosis data 440 asdiscussed herein. While not shown in FIG. 18C, this second set ofannotation data can be sent to the medical scan database for storage inconjunction with the second set of medical scans and/or can betransmitted to other subsystems 101 for use.

In particular, this annotation data can be utilized to aid users ingenerating additional annotation data via interaction with interactiveinterface 7075, as illustrated in FIG. 18C. The annotation data can bedisplayed in conjunction with a corresponding medical scan viainteractive interface 7075. For example, a first client device candisplay a medical scan 2.1 of the second set of medical scans inconjunction with the corresponding annotation data 2.1 generated by themodel-assisted annotating system 7002. The annotation data 2.1 can bedisplayed as text and/or can be utilized to visually indicate one ormore abnormalities detected in the medical scan 2.1.

Some or all of the annotation data can be displayed visually, overlayingimage data of the medical scan displayed by the interactive interface7075. For example, the annotation data 2.1 can include localization dataindicating a region of interest where an abnormality was detected, forexample, based on probability matrix data. This localization data caninclude segmenting data automatically generated by the model-assistedannotating system 7002, and the segmenting data can be utilized todisplay a polygon shape outlining and/or surrounding a detectedabnormality indicated in the annotation data, overlaying image data ofthe medical scan displayed by the interactive interface 7075. Theannotation data can include the heat map visualization data of FIG. 14C,which can be displayed in conjunction with the medical scan via theinteractive interface 7075. The annotation data can include measurementdata indicating a diameter, area, and/or volume automatically calculatedfor an abnormality detected in the medical scan, and the measurementdata can be displayed upon an abnormality detected in the image data.For example, a diameter measurement can be displayed upon an abnormalitydetected in the image data.

Alternatively or in addition, some or all of the annotation data 2.1 canbe displayed by automatically making one or more prompt selections asdiscussed in conjunction with the medical scan hierarchical labelingsystem. In particular, one or more of the hierarchical interfacefeatures, such as those illustrated in FIGS. 13S-13V, can beautomatically selected, where checked boxes or other visual indicationsof selection automatically appear based on corresponding labels beingindicated in the annotation data 2.1. Furthermore, the prompt decisiontree can automatically progress to an internal node prompt based on oneor more decisions already made as indicated in the annotation data. Forexample, a user can be automatically presented with an interface asdisplayed in FIG. 13T in response to the automatically generatedannotation data 2.1 indicating that the medical scan includes anabnormality characterized as pulmonary vasculature, plethora, anddiffuse, where these selections are already visually indicated, andwhere the user is automatically prompted to select from “left”, or“right.”

Client devices can generate additional annotation data based on userinput to the interactive interface 7075. The generation of theadditional annotation data can be assisted based on thealready-generated annotation data presented to the user via theinteractive interface 7075. The additional annotation data can includecorrection and/or edits to the annotation data, indicating that some ofthe annotation data is incorrect. Alternatively, the additionalannotation data can indicate all of the annotation data is correct, andcan include further detail, for example, to further characterize and/orlocalize an abnormality indicated in the annotation data. The additionalannotation data can indicate both the originally generated annotationdata as well as new data inputted by the user, and/or can include onlyadditional information and/or corrections made by the user.

For example, the interactive interface 7075 can automatically outline,highlight, zoom-in on, or otherwise visually indicate at least oneautomatically detected abnormality indicated in the annotation data. Theinteractive interface 7075 can prompt the user to confirm whether or noteach detected abnormality is indeed an abnormality and/or to confirmand/or edit a type of abnormality determined for the detectedabnormality. The interactive interface 7075 can prompt the user to editany automatically generated characterization data of each abnormality,for example, via interaction with a text box and/or by allowing the userto de-select any automatically made selections, such as automaticallydetermined selections of a hierarchical labeling display such as that ofFIGS. 13S-13V.

The interactive interface 7075 can prompt the user to provide additionalcharacterization data for an abnormality indicated in the image data.For example, the user can be automatically presented with an interfaceas displayed in FIG. 13T in response to the automatically generatedannotation data 2.1 indicating that the medical scan includes anabnormality characterized as pulmonary vasculature, plethora, anddiffuse, as discussed. The user can then be prompted to make finalselections until a leaf node of the prompt decision tree is reached, forexample, where the final selections of “left” and “lobe—left upper” areselected via user input as illustrated in FIG. 13U. These selections canbe included in the additional annotation data.

The user can be prompted to provide and/or edit measurement data for adetected abnormality, where a user is prompted to draw a diameter for alesion presented to the user, indicate endpoints of a diameter of alesion presented to the user, and/or otherwise provide measurement datafor a lesion presented to the user. The user can be prompted to provideand/or edit segmentation data for the detected abnormality, where theuser is prompted to draw and/or indicate vertices for a polygonsurrounding the detected abnormality. These selections can be includedin the additional annotation data.

The user can be further prompted to identify, characterize, and/orlocalize any additional abnormalities that were not indicated in theannotation data. These selections can be included in the additionalannotation data.

The additional annotation data 2.1-2.N generated for each of the medicalscans in the second set of medical scans can be transmitted to themedical scan database for storage in conjunction with the correspondingmedical scans. Furthermore, as illustrated in FIG. 18D, this second setof additional annotation data can be utilized to update the currentmodel and/or to update a plurality of sub-models. Training step 7012 canbe performed and can include retraining the current model based oncorrected annotation data of the additional annotation data, forexample, by utilizing the weights and/or other parameters of the currentmodel and by retraining and/or overfitting the current model to modifythe current set of weights to generate an updated model and/or anupdated plurality of sub-models. Image data of the second set of medicalscans can be utilized as input data and the additional annotation datacan be utilized as output labels to retrain and/or fine-tune the currentmodel. Some or all of the automatically generated annotation data canalso be included in the output labels, for example, for medical scans ofthe second set with automatically generated annotation data determinedto be correct and/or complete. The updated model can correspond to oneor more new, fine-tuned models generated by fine-tuning an existingmodel as discussed in conjunction with FIG. 16B.

Parameters, weights and/or other data for the updated model and/orupdated plurality of sub-models can be transmitted to the medical scananalysis function database 346 for use by other subsystems 101. Theupdated model can also be stored locally by the model-assistedannotating system for further use, as illustrated in FIG. 18E.

The updated model can correspond to a more accurate version and/or morecomprehensive of the current model, trained on additional training datathat can include more types of medical scans and/or types ofabnormalities not sufficiently included in the original training data ofthe first set of medical scans and the first set of annotation data.Furthermore, the automatic assistance provided via interactive interface7075 to generate the additional annotation data can increase accuratedetection of abnormalities, leading to more accurate annotation dataprovided by users to generate a more accurate model. The assistanceprovided via interactive interface 7075 to generate the additionalannotation data can increase efficiency, allowing users to more quicklyannotate medical scans and leading to a greater volume of annotationdata provided by users, allowing for larger training sets to generate amore comprehensive model.

In some embodiments, the current model generates a first type of output,such as abnormality detection data with no characterization data. Theupdated model can utilize additional information provided in the set ofadditional annotation data to improve the functionality of the currentmodel, for example, where a second type of output is utilized to trainthe updated model, where the second type of output includescharacterization data and/or other new output labels that were notincluded in the current model. Thus, the updated model can provideadditional functionality, where output of the corresponding inferencefunction included additional probabilities for additional labels notincluded in the current model, and/or otherwise generates additionalinformation not generated by the current model. This can includegenerating one or more new types of models, such as one or more of thesub-models 1-K that were not previously available, based additionaltypes of medical scans and/or abnormality data included in the secondset of medical scans and/or second set of additional abnormality data.This can also include generating one or more further-fine-tuned models,for example, increasing functionality of an existing fine-tuned-model bytraining on more detailed characterization and/or localization labelsindicated in the additional annotation data to generate a model that canprocess medical scans and can similarly generate more detailedcharacterization and/or localization labels.

As illustrated in FIG. 18E, a third set of medical scans can be receivedvia the network, and a corresponding third set of annotation data can begenerated by utilizing an inference function 7022. The inferencefunction 7022 can utilize the updated model generated by performingtraining step 7012 in FIG. 17D. Inference function 7022 can utilizeinference step 6030 to perform one or more inference functions 1-Jselected for the medical scan and/or a final inference function, wherethe annotation data corresponds to the final abnormality data. Inferencefunction 7022 can utilize inference function 5020, global labelingfunction 5030, and/or heat map generator function 5040, where theannotation data includes the probability matrix data, global probabilitydata, and/or heat map visualization data discussed in conjunction withFIGS. 14B and 14C. Inference function 7022 can utilize inference step1354, detection step 1372, abnormality classification step 1374, similarscan identification step 1376, and/or display parameter step 1378 asdiscussed in conjunction with FIG. 7B.

The third set of annotation data can be transmitted to the medical scandatabase 342 as illustrated in FIG. 18E, for example, in response to theupdated model being determined to be complete and/or sufficientlyaccurate. Alternatively, the process illustrated in FIGS. 18C and 18Dcan repeat for this third set of medical scans, where the third set ofmedical scans and corresponding set of annotation data are sent to thesame or different set of client devices, and where additional annotationdata is generated by the client devices based on user input theinteractive interface 3075 in response to a prompt displayed via theinteractive interface 3075 in conjunction with display of one or more ofthe third set of medical scans and corresponding set of automaticallygenerated annotation data. This third set of additional annotation datacan be utilized to further update the current model and/or one or moreof a plurality of sub-models.

This process can continue to repeat for subsequent sets of medicalscans, where the model and/or a plurality of sub-models are furtherupdated over time to improve accuracy and/or increase functionality. Themodel-assisted annotating system 7002 can automatically determine whenthis process should be repeated, for example, in response to determininga generic model and/or one or more sub-models are not sufficientlyaccurate, do not generate enough information to fully characterize oneor more types of abnormalities, and/or were not trained on a sufficientamount of training data corresponding to one or more specific types ofmedical scans and/or corresponding to one or more specific types ofabnormalities, resulting in the generic model and/or one or moresub-models not sufficiently handling the one or more specific types ofmedical scans and/or not sufficiently detecting and/or characterizingthe one or more specific types of abnormalities. Retraining one or moremodels as discussed in conjunction with FIGS. 18C and 18D can beincluded in a remediation step 1140, in response to determining totemporarily decommission one or of the models and/or in response todetermining the one or more models needs to be retrained by utilizingnew training data.

In some embodiments, subsequent rounds of training include increasingcharacterization functionality, for example, based on more and moredetailed characterization indicated by the characterization promptdecision tree and/or another prompt decision tree. In particular, earlyrounds of training can automatically generate detection data and/or lessdetailed characterization data corresponding to internal nodes of thecharacterization prompt decision tree that are close to the root node.The additional annotation data can include standardized labeling datacorresponding to one or more deeper layers of the prompt decision treeand/or leaf nodes. The training step 7012 can include fine-tuning anexisting model trained to characterize a particular type by introducingadditional output labels corresponding to selections made in a deeperlayer of the prompt decision tree. This process can repeat as additionaltraining data is available to enable automatic characterizationcorresponding to deeper and deeper layers of the prompt decision tree,for example, where corresponding further-fine-tuned models are trainedby overfitting a model trained in a previous iteration, based on theadditional annotation data, for enable even more characterizationfunctionality. This process can similarly be implemented forlocalization by utilizing the localization prompt decision tree.

The model-assisted annotating system 7002 can further determine whattype of training data is necessary for subsequent rounds of updatedtraining, and can request specific types of medical scans from themedical scan database by automatically generating medical scan criteriaindicating particular scan classifier data 420 such as one or moreparticular modalities, anatomical regions, sequences, geographicregions, hospital settings, and/or originating entities. Furthermore,for medical scans that may already have diagnosis data 440 in themedical scan database generated by a client device based on user inputand/or automatically generated by another subsystem 101, the medicalscan criteria can further indicate particular diagnosis data 440,indicating particular types of abnormalities and/or particularabnormality classification data 445 for one or more abnormalityclassifier categories 444 and/or abnormality pattern categories 446. Aset of medical scans corresponding to this automatically generatedmedical scan criteria can be automatically retrieved from the medicalscan database 342. A set of annotation data can be automaticallygenerated utilizing one or more existing models, and this annotationdata can be sent to the same or different set of client devices with thecorresponding set of medical scans. The additional annotation data canbe generated as discussed in conjunction with FIG. 18C. A generic modeland/or one or more sub-models can be automatically updated by performingthe same or different training step 7012 to increase accuracy of one ormore existing models, to improve functionality of one or more existingmodels, and/or to create one or more new models, for example, configuredto handle medical scans and/or abnormalities identified in theautomatically generated medical scan criteria.

Generating the medical scan criteria can include identifying weakcategories based on evaluating the annotation data generated across aset of medical scans. These identified weak categories can be utilizedto automatically dictate types of training data that should beprioritized to improve the model, and can be used to identify additionalscans to include in the training data to train an improved model.

For example, the automatically generating medical scan criteria can begenerated for subsequent rounds of updated training based onautomatically detecting weak categories of medical scans. This caninclude evaluating a set of annotation data generated for a set ofmedical scans on a most current model and/or a most current one or moresub-models. Confidence levels and/or a confusion matrix of theannotation data for the set of scans can be compared to thresholdsacross multiple scan classifier categories to identify weak categories.This can include utilizing the probabilities indicated in probabilitymatrix data generated by performing an inference step 1354 and/orinference function 5020.

A weak scan classifier categories can include medical scans of with scanclassifier data 420 or one or more particular categories where athreshold number the medical scans in the identified scan classifiercategory have corresponding annotation data that has confidence levelsconfusion matrices that compare unfavorably to the threshold, forexample, indicating abnormalities are identified with weak levels ofconfidence. Similarly, additional annotation data received from clientdevices indicating a threshold number of medical scans of a particularscan classifier category with incorrect annotation data can be utilizedto identify the particular scan classifier category as a weak scanclassifier category. This weak scan classifier category can be indicatedin the medical scan criteria, and a set of medical scans correspondingto the weak scan classifier category can be received in response fortransmission to client devices with or without correspondingautomatically generated annotation data. This set of medical scans andthe additional annotation data received from the set of client devicescan be utilized to perform training step 7012 to generate updated and/ornew sub-models and/or to generate an updated generic model.

As another example, the automatically generating medical scan criteriacan be generated for subsequent rounds of updated training based onautomatically detecting weak categories of output labels. This caninclude evaluating a set of annotation data generated for a set ofmedical scans on a most current model and/or a most current one or moresub-models. Confidence levels and/or a confusion matrix of theannotation data for the set of scans can be compared to thresholdsacross multiple output label categories to identify weak categories. Aweak output label category can correspond to an output label withcorresponding confidence levels and/or corresponding probability valuesin a probability matrix that compare unfavorably to the threshold,indicating these output labels are identified with weak levels ofconfidence. For example, one or more types of abnormalities of amulti-label model generated as discussed in conjunction with themulti-label medical scan analysis system can be identified as weak whena threshold number of abnormality data indicates the correspondingoutput labels have corresponding probabilities that compare unfavorablyto the threshold.

Similarly, additional annotation data received from client devicesindicating a threshold number of automatically generated annotation dataindicating the type of abnormality was incorrect can be utilized toidentify the type of abnormality as a weak type of abnormality. The weakoutput label can be indicated in the medical scan criteria, and a set ofmedical scans corresponding to the weak output label can be received inresponse. For example, medical scans with already-generated diagnosisdata 440 matching the weak output label can be retrieved. This set ofmedical scans can be sent to client devices to allow thealready-generated diagnosis data 440 to be confirmed and/or edited inthe additional annotation data, and this additional annotation data canbe received from the set of client devices can be utilized to performtraining step 7012 to generate updated and/or new sub-models and/or togenerate an updated generic model.

Alternatively, the set of medical scans and the already-generateddiagnosis data 440 corresponding to the weak label can be utilizedperform training step 7012 without the need for additional annotationdata.

Other problematic trends across the automatically generated annotationdata can be identified based on a set of rules. For example, detectingan instance or trend that the global abnormal score is high where noindividual abnormality pattern is high can be used to flag a problemwith the model. Similarly, detecting an instance or trend where theglobal abnormality score is low and one or more individual scores ishigh can also be used to flag a problem with the model. As anotherexample, the set of rules can include sets of findings with requireddependencies, indicating sets of findings that should or shouldn't occurin tandem. Detecting an instance or trend that one of these sets offindings occurs in tandem can be used to flag a problem with the model.These identified problematic trends can also be used to automaticallydictate types of training data that should be prioritized to improve themodel.

When one of the set of rules is broken by a threshold number of the setof annotation data, retraining can be determined to be necessary. If oneof more of the set of rules is broken by a threshold number of the setof annotation data for scans of a particular type, medical scans withthe corresponding scan classifier data for these scans can be requestedas additional training data. one or more particular output labels aredetermined to be included in a set of required dependencies that isbroken by a threshold number of the set of annotation data, additionalmedical scans with these output labels can be requested as additionaltraining data.

A set of required dependencies in the set of rules can indicate outputlabels for findings that should occur in tandem, and this rule can bedetermined to be broken for annotation data with one of the outputlabels of the required dependency having a corresponding probability ofbeing present in the medical scan that compares favorably to a detectionprobability threshold, and with a second one of the output labels of therequired dependency having a corresponding probability of being presentin the medical scan that compares unfavorably to the same or differentdetection probability threshold. The set of required dependencies canindicate output labels for findings that should not occur in tandem, andthis rule can be determined to be broken for annotation data with two ormore of the output labels of the required dependency havingcorresponding probabilities of being present in the medical scan thatcompare favorably to the same or different detection probabilitythresholds. Alternatively or in addition, determining whether or notfindings have occurred in tandem in the annotation data can includecalculating a difference between detection probabilities of thecorresponding findings, where the findings are determined not to occurin tandem if the difference between the detection probabilities exceedsa difference threshold.

The annotation data can indicate a global abnormality probability,indicating a probability that any abnormality is present. The annotationdata can further include a plurality of abnormality type probabilities,each indicating a probability that one of a set of correspondingabnormality types is present. The set of rules can dictate that when theglobal probability compares favorably to a detection probabilitythreshold, then at least one of the abnormality type probabilities mustcompare favorably to the same or different detection probabilitythreshold. The set of rules can dictate that when at least one of theabnormality type probabilities compares favorably to a detectionprobability threshold, then the global probability must comparefavorably to the same or different detection probability threshold. Theset of rules can dictate that when the global probability comparesunfavorably to a detection probability threshold, then all of theabnormality type probabilities must compare unfavorably to the same ordifferent detection probability threshold. The set of rules can dictatethat when all of the abnormality type probabilities compare unfavorablyto a detection probability threshold, then the global probability mustcompare unfavorably to the same or different detection probabilitythreshold.

The set of rules can be generated based on user input to an interactiveinterface displayed via a display device in response to a prompt toindicate the set of rules. Alternatively or in addition, the set ofrules can be generated based on automatically determining trends intraining data, such as automatically identifying findings that always oroften occur in tandem and/or automatically identifying findings thatrarely or never occur in tandem, and/or otherwise automaticallydetermining pair or other sets of findings that have a high correlationthat is determined to be statistically significant.

In various embodiments, a model-assisted annotating system 7002 includesat least one processor and a memory that stores operational instructionsthat, when executed by the at least one processor, cause themodel-assisted annotating system to receive, via a receiver, a first setof medical scans for review. The first set of medical scans can betransmitted, via a transmitter, to a set of client devices associatedwith a set of users. Each of the first set of medical scans can bedisplayed to one of the set of users via an interactive interfacedisplayed by a display device associated with a corresponding one of theset of client devices. A first set of annotation data can be receivedfrom the set of client devices. Each of the first set of annotation datacan be generated by one of the set of client devices in response to aprompt via the interactive interface displayed by the display device toprovide annotation data for one of the first set of medical scansdisplayed by the one of the set of client devices.

A first training step can be performed to train a computer vision modelby utilizing image data of the first set of medical scans as inputlabels and by utilizing the first set of annotation data as outputlabels. A second set of medical scans can be received, via the receiver,for review. A second set of annotation data can be generated byperforming an inference function on image data of the second set ofmedical scans. The inference function can utilize the computer visionmodel. Each of the second set of annotation data can be generated asoutput of performing the inference function on a corresponding one ofthe second set of medical scans.

The second set of medical scans and the second set of annotation datacan be transmitted, via the transmitter, the to the set of clientdevices, wherein each of the second set of medical scans is displayed toone of the set of users in conjunction with a corresponding one of thesecond set of annotation data via the interactive interface. A set ofadditional annotation data can be received from the set of clientdevices. Each of the set of additional annotation data can be generatedby one of the set of client devices in response to a prompt via theinteractive interface displayed by the display device to provideadditional annotations to the corresponding one of the second set ofannotation data displayed in conjunction with one of the second set ofmedical scans by the one of the set of client devices. A second trainingstep can be performed to generate an updated computer vision model byutilizing the set of additional annotation data to update the computervision model.

A third set of medical scans can be received, via the receiver, forreview. A third set of annotation data can be generated by performing anupdated inference function on image data of the third set of medicalscans. The updated inference function can utilize the updated computervision model, and each of the third set of annotation data can begenerated as output of performing the updated inference function on acorresponding one of the third set of medical scans. The third set ofmedical scans and the third set of annotation data can be transmitted,via the transmitter, to the set of client devices. Each of the third setof medical scans can be displayed to one of the set of users inconjunction with a corresponding one of the third set of annotation datavia the interactive interface.

FIG. 19 presents a medical scan triaging system 8002. The medical scantriaging system 8002 can be utilized to automatically foregohuman-review of medical scans that are determined to be normal with ahigh enough probability. In particular, human-review of these medicalscans can be waived based on determining global abnormality thresholdsgenerated by performing an inference function on these medical scanscompares unfavorably to a triage probability threshold, where the triageprobability threshold is generated automatically or generated viaadministrator input. The remaining medical scans with a high enoughprobability that some abnormality exists can be triaged for review by aplurality of users via interaction with an interactive interfacedisplayed via display devices of a plurality of client devices. Inparticular, the remaining medical scans are designated for human reviewbased on determining global abnormality thresholds generated byperforming an inference function on these medical scans comparesfavorably to a triage probability threshold. These triaged scans can bequeued in a designated order and/or can be designated for particularusers based on the type of scan, based on types of abnormalitiesdetected, and/or based on a determined severity, rarity, and/ortime-sensitivity of detected abnormalities.

As shown in FIG. 19, the medical scan triaging system 8002 cancommunicate bi-directionally, via network 150, with the medical scandatabase 342 and/or with other databases of the database storage system140, with one or more client devices 120, and/or, while not shown inFIG. 19, with one or more subsystems 101 of FIG. 1, with the centralserver system 2640, and/or with the medical picture archive system 2600.

In some embodiments, the medical scan triaging system 8002 is anadditional subsystem 101 of the medical scan processing system 100,implemented by utilizing the subsystem memory device 245, subsystemprocessing device 235, and/or subsystem network interface 265 of FIG.2A. For example, the medical scan triaging system 8002 can beimplemented by utilizing the medical scan image analysis system 112, themulti-label medical scan analysis system 5002, the multi-model medicalscan analysis system 6002, and/or the model-assisted annotating system7002 to train one or more computer vision models and/or to perform oneor more inference functions by utilizing the one or more computer visionmodels. Furthermore, the generation of human-generated annotation datavia user input to interactive interfaces by utilizing client devices120, as illustrated in FIG. 19, can be implemented by utilizing themedical scan assisted review system 102, the medical scan annotatingsystem 106, the medical scan hierarchical labeling system 3002, and/orthe model-assisted annotating system 7002.

In some embodiments, the medical scan triaging system 8002 utilizes, orotherwise communicates with, the central server system 2640. Forexample, the medical scan database 342 can be populated withde-identified data generated by the medical picture archive integrationsystem 2600. The medical scan triaging system 8002 can receivede-identified medical scans of with their corresponding annotation data,diagnosis data, and/or medical reports directly from the medical picturearchive integration system 2600, for example, where the annotation data,diagnosis data, and/or medical reports are utilized to determine themedical labels for medical scans in a training set utilized to train oneor more inference functions utilized by the medical scan triaging system8002. As another example, the medical scan triaging system 8002 canperform one or more inference functions on de-identified medical scansreceived from the medical picture archive integration system 2600, andabnormality data or other inference data generated as output of theplurality of the one or more inference functions can be assigned to themedical scan in the medical picture archive integration system 2600. Asanother example, the medical scan triaging system 8002 can requestde-identified medical scans, annotation data, and/or reports that matchrequested criteria medical scans to be triaged. In some embodiments,some or all of the medical scan triaging system 8002 is implemented byutilizing other subsystems 101 and/or is operable to perform functionsor other operations described in conjunction with one or more othersubsystems 101.

As illustrated in FIG. 19, a set of medical scans 1-N can be received bythe medical scan triaging system 8002. For example, these medical scanscan correspond to new medical scans to be reviewed, and can betransmitted to the medical scan triaging system 8002 as they arecaptured by imaging machines and/or can be requested by the medical scantriaging system 8002. The medical scan triaging system 8002 can performan inference function 8010 to generate inference data for the pluralityof medical scans 1-N. The inference function utilize a computer visionmodel trained on training set of medical scans and correspondinglabeling data. For example, the corresponding labeling data utilized totrain the inference function 8010 can include a global abnormality labelindicating whether each scan in the training set is normal or includesat least one abnormality. As another example, the inference function8010 can utilize inference function 7020, inference function 7022,inference step 6030, inference function 5020, inference step 1354,and/or detection step 1372 to generate inference data.

The inference data generated by the inference function 8010 can includea global abnormality probability indicating whether any abnormality ispresent. This global abnormality probability can be determined byperforming the final inference function of FIG. 16C based on abnormalitydata of a plurality of sub-models performed on each medical scan. Theglobal abnormality probability can be generated based on evaluatingwhether any of a plurality of output labels generated as output to theinference function 8010 indicate that a corresponding one of a pluralityof abnormalities is present, for example, by utilizing one or moreprobability matrices 1371 and/or probability matrix data generated byutilizing inference function 5020. For example, the global abnormalityprobability can be generated by utilizing the global probability datagenerated by performing the global labeling function 5030 on theprobability matrix data generated by performing inference function 5020.

In some embodiments, the inference data for a medical scan includes onlyone output label, indicating a single probability that any abnormalityis present. In other embodiments, the inference data for a medical scanincludes a plurality of probabilities that a plurality of abnormalitytypes are present, and the global abnormality probability is generatedas a function the plurality of probabilities of the plurality ofabnormality types. In some embodiments, the inference data for a medicalscan includes a plurality of probabilities corresponding to a pluralityof regions of the image data of the medical scan, where the globalabnormality probability is generated as a function the plurality ofprobabilities of the plurality of regions.

The global abnormality probability can indicate a highest probabilityacross a plurality of probabilities that each of a plurality ofabnormality types are present. The global abnormality probability canindicate a higher probability than any of the plurality of probabilitiesthat each of the plurality of abnormality types are present in responseto determining that multiple ones of the plurality of abnormality typeshave a probability of being present in the medical scan that are higherthan a detection probability threshold. The global abnormalityprobability can indicate a higher probability than any of a plurality ofprobabilities that each of a plurality of abnormalities are present inresponse to determining that multiple abnormalities are detected indifferent regions of the image data, each with a probability of beingpresent in the medical scan that is higher than a detection probabilitythreshold.

In some embodiments, when probability matrix data indicates a pluralityof probabilities that each of a plurality of regions of image data of amedical scan include a probability, the global abnormality probabilitycan be higher in response to a higher number of adjacent regionsindicating a probability that an abnormality exists that is higher thana detection probability threshold. For example, the global abnormalityprobability can be higher than any of the probabilities of the pluralityof adjacent regions, and the increase in value of the global abnormalityprobability from an average or highest probability of the adjacentregions can be an increasing function of the number of adjacent regions.

In some embodiments, when a single region indicates a probability thatan abnormality exists that is higher than a detection probabilitythreshold, surrounded by adjacent regions that are substantially lowerthan the detection probability threshold, the global abnormalityprobability is generated to be lower than the probability in the singleregion. In some embodiments, a filtering and/or smoothing step can beapplied to probability matrix data generated as output to the inferencefunction 8010, where the probabilities of these single regions areconsidered extraneous probabilities that are removed and/or lowered, andwhere the global abnormality probability is generated to indicate aprobability that an abnormality is present that is lower than thedetection probability threshold despite these extraneous probabilitiesof the probability matrix data.

While the global abnormality probability discussed herein indicates avalue that increases with probability that an abnormality is present,the global abnormality probability can instead be represented as aglobal normal probability. The value of the a global normal probabilitycan be equal to one minus the global abnormality probability, and canindicate a value that increases with probability that no abnormality ispresent. The inference data generated as output to the inferencefunction 8010 can indicate either the global abnormality probability orthe global abnormality probability.

A triage assignment function 8020 can be performed once the globalabnormalities are generated for the set of medical scans 1-N. The triageassignment function 8020 can group the medical scans into two groups: anormal group of medical scans determined to be normal, where review by ahuman is determined not to be necessary, and a triaged group of medicalscans, where review by a human review is determined to be necessary.Dividing the medical scans into the two groups can be performed bycomparing the global abnormality probability to a triage probabilitythreshold. Medical scans with global abnormality probabilities thatcompare favorably to the triage probability threshold can be included inthe triaged group, and medical scans that compare unfavorably to thetriage probability threshold can be included in the normal group. Thetriage probability threshold can be determined automatically by themedical scan triaging system 8002 and/or can be determined based onreceiving a triage probability threshold from a client device 120 basedon user input to an interactive interface 8075 displayed on a displaydevice, in response to a prompt to select a triage probability thresholddisplayed on the interactive interface 7075. For example, anadministrator can set the triage probability threshold, and can changethe triage probability threshold to a higher or lower value at a latertime.

The triage probability threshold can correspond to a very strictstandard, for example, where medical scans must have a global normalprobability of at least 99% or other substantially high probability thatno abnormalities exist to be placed in the normal group. The triageprobability threshold can be the same or different from any of thedetection probability thresholds discussed herein. The triageprobability threshold can be the same or different for different typesof medical scans and/or for different types of abnormalities, and aplurality of triage probability thresholds for these different groupscan be individually set via user input to interactive interface 8075.

Medical scans assigned to the triage group can be automatically assignedto one or more particular users for review, for example based onperformance data and/or qualification data for these users in the userdatabase. In particular, users can be selected as discussed inconjunction with the medical scan annotation system 106, for example,based on the type of medical scan, based on one or more types ofdetected abnormalities in the inference data, and/or based on an urgencyvalue in the inference data indicating a time-sensitivity of the one ofmore detected abnormalities. Alternatively or in addition, the users beselected based on availability of the users, based on a queue size ofother scans triaged to the users, and/or based on a random orpseudo-random selection.

The corresponding client devices 120 can be determined based on theselected users, and the medical scan triaging system 8002 can facilitatetransmission of each medical scans in the triage group to one or morecorresponding client devices 120 based on determined assignments.Alternatively, the medical scan triaging system 8002 can transmit triagedata to the medical scan database 342 and/or another entity indicatingidentifiers of the medical scans in the triage group, and the medicalscans in the triage group can be retrieved from or automaticallytransmitted to the client devices 120 accordingly. For example, asillustrated in FIG. 19, medical scans 1-R assigned to the triage groupcorrespond to a proper subset of the set of medical scans 1-N.Identifiers indicating the medical scans medical scans 1-R in the triagegroup can be transmitted via the network, for example, the medical scandatabase. Each of the medical scans 1-R can be sent to one or morecorresponding client devices 120 for review from the medical scandatabase 342 and/or from the medical scan triaging system 8002.Alternatively, the identifiers indicating the medical scans 1-R and/or asubset of the identifiers indicating a subset of the triaged scansassigned to each user can be sent to each corresponding client device,and the client devices can retrieve triaged medical scans accordingly byutilizing the corresponding identifiers.

The medical scans in the normal group can identified as not needingadditional review, and identifiers indicating the medical scans in thenormal group can similarly be sent to the medical scan database 342and/or another entity. For example, these medical scans can be flaggedas normal, can be flagged as review already having been completed,and/or can be flagged as not needing review in a corresponding entry ofthe medical scan database. For example, as illustrated in FIG. 19,medical scans 1-Q assigned to the normal group correspond to a propersubset of the set of medical scans 1-N. The set of scans 1-R and the setof scans 1-Q can be mutually exclusive and collectively exhaustive withregards to the set of scans 1-N. Identifiers indicating the medicalscans medical scans 1-Q in the normal group can be transmitted via thenetwork, for example, the medical scan database. Alternatively or inaddition, these medical scans can be removed from a set of scanspreviously designated for triage. Alternatively or in addition, theidentifiers indicating the medical scans in the normal group can be sentto one or more client devices, where users are alerted that these scansdo not need review via a display on interactive interface 8077.

In some embodiments where only the global abnormality probability isindicated in the output of inference function 8010, additional inferencefunctions can be applied to medical scans in the triage group. Forexample, once a medical scan is determined to be designated as needingfurther review, the sub-model selection step 6020 can be applied to themedical scan, and one or more selected sub-models 1-J can be applied togenerate additional inference data by applying inference step 6030. Insome embodiments, the additional inference data can be utilized fordetermining which user will be selected for triage, for example, byutilizing additional details such as a type of abnormality and/orlocalization data for an abnormality to select a user with correspondingqualifications. The additional inference data can indicatetime-sensitivity data, which can be utilized to determine to select auser that will be available to review the scan or see the patient inperson more quickly. The additional inference data can indicatecorresponding probabilities for some or all of this information, andmore qualified users can be selected in response to determining anabnormality's presence in the image data is more uncertain, and thatmore scrutiny will be necessary to review the scan. Similarly, lessqualified users can be selected to review a medical scan in response todetermining an abnormality's presence in the image data is more certainand/or is characterized with more detail and/or with higher confidence,where less scrutiny may be required to review the scan. Some or all ofthis additional information provided by performing subsequent inferencefunctions can instead be indicated in the original inference datagenerated as output of inference function 8010.

Interactive interface 8077 can display medical scans triaged to thecorresponding client device via a corresponding display device to theuser. Interactive interface 8077 can allow the user to enter annotationdata, report data, diagnosis data, or other data describing anyabnormalities determined to be included in the medical scans, forexample by utilizing interface features of the medical scan assistedreview system 102 and/or the medical scan hierarchical labeling system3002. Some or all of the additional information generated by theinference function 8010 and/or any additional inference functionsperformed can be transmitted in conjunction with medical scans to clientdevices 120 for display to assist users in reviewing the medical scans,for example, as discussed in conjunction with the medical scan assistedreview system 102 and/or as discussed in conjunction with themodel-assisted annotation system 7002. Alternatively, the medical scanscan be reviewed in their original form. The annotation data generatedfor medical scans reviewed by a user of a client device 120 can betransmitted to the medical scan database for storage, for example, assome or all of diagnosis data 440. As shown in FIG. 19, each annotationdata 1-R, corresponding to the triaged medical scans 1-R, can betransmitted from the client devices 120 to the medical scan database 342for storage, for example, as some or all of diagnosis data 440. Eachannotation data 1-R can include structured labeling data, region ofinterest data, unstructured text, and/or any other diagnosis data 440 asdiscussed herein.

When the triage probability threshold is changed automatically or basedon user input to interactive interface 8075, subsequently receivedmedical scans can be assigned to the triage group or the normal group inaccordance with the new triage probability threshold. Furthermore, someof the already-sorted medical scans 1-N can be switched to a differentgroup accordingly. For example, when the triage probability thresholdbecomes stricter, requiring a higher global normal probability for scansto be placed in the normal category, one or more medical scans includedin the normal category because they compared unfavorably to a priortriage probability threshold can be assigned for triage in response tocomparing favorably to the new, stricter triage probability threshold.These medical scans can be similarly designated for review by one ormore particular users and/or can be queued for review. The medical scandatabase can be updated to change the designation of these scans fromnormal to needing review, until review is complete.

When the triage probability threshold becomes looser, requiring a lowerglobal normal probability for scans to be placed in the normal category,one or more medical scans included in the triage category because theycompared favorably to a prior triage probability threshold can beassigned as normal in response to comparing unfavorably to the new,looser triage probability threshold. If these medical scans were alreadytransmitted for review to one or more client devices 120, they can beremoved from a queue of scans to review and/or a notification can betransmitted to the client device 120 to indicate these medical scans nolonger require review. Identifiers for these medical scans can betransmitted to the medical scan database, and these medical scans can beflagged as normal and/or as not requiring review in the medical scandatabase.

The triage probability threshold can automatically be adjusted by themedical scan triaging system 8002 at any time. For example, the medicalscan triaging system 8002 can determine to change the triage probabilitythreshold to a stricter triage probability threshold in response todetermining that the accuracy of the inference function 8010 hasworsened and/or compares unfavorably to a threshold. The medical scantriaging system 8002 can determine to change the triage probabilitythreshold to a stricter triage probability threshold in response todetermining an expected average review time compares favorably to a lowreview time threshold, and thus more medical scans can be designated forreview without overloading users in the system and/or without delayingthe expected average review time to compare unfavorably to the lowreview time threshold.

The medical scan triaging system 8002 can determine to change the triageprobability threshold to a looser triage probability threshold inresponse to determining the accuracy of the inference function 8010 hasimproved and/or compares favorably to a threshold. The medical scantriaging system 8002 can determine to change the triage probabilitythreshold to a looser triage probability threshold in response todetermining the expected average review time to compares unfavorably tothe low review time threshold to ensure that medical scans are beingreviewed more quickly.

Performing the triage assignment function 8020 can include generatingpriority data for the medical scans in the triage group, for examplewhere medical scans are designated for a review in an order designatedby a plurality of priority values generated for the plurality of medicalscans. The priority values be a function of global abnormalityprobability, and can be assigned a higher values for higherprobabilities that an abnormality exists, as dictated by the globalabnormality probability. The priority values can be function of aseverity level and/or rarity level corresponding to one or more types ofabnormalities detected in the medical scan, where medical scans withdetected types of abnormalities determined to be more rare or moresevere are assigned higher probability values. The priority values canbe function of a global time-sensitivity value indicated in theinference data generated by performing inference function 8010, forexample, where the time-sensitivity value is based on one or more outputlabels in the inference data, and where the inference function 8010 wastrained by utilizing output labels for the training set of medical scansthat indicated one or more time-sensitivity values based on andetermined time-sensitivity for the training set of medical scans. Thepriority value can be assigned a higher value for highertime-sensitivity values. The priority value can be determined as aweighted sum, weighted average, or other function of multiple criteriaincluding one or more of the global abnormality probability, severityvalue, rarity value, time-sensitivity value, and/or other output labelsindicated in the inference data.

The medical scans can be queued for review in an order dictated by thecorresponding priority values, where medical scans with the highestpriority values are queued for review first. Selecting which users willreview each medical scan can further be based on the ordering ofpriority values, where users determined to be available to reviewmedical scans more quickly and/or more accurately are selected to reviewmedical scans with higher priority values, and/or where a plurality ofmedical scans with highest priority values are divided amongst multipleusers, ensuring each of these users can review these highest prioritymedical scans first and ensuring that all of the highest priority scansare reviewed in a timely fashion. A subset of medical scans designatedfor a particular user can be queued for that user in accordance with thepriority values, where the interactive interface 8077 presents themedical scans to the user one at a time for review in accordance withthe ordering dictated by the priority values.

In some embodiments, some medical scans with global abnormalityprobabilities that compare unfavorably to the triage probabilitythreshold can still be designated for triage. For example, if aseverity, rarity, and/or time-sensitivity value determined in theinference data for a medical scan exceeds a corresponding severity,rarity, and/or time-sensitivity threshold, these medical scans cansimilarly be designated for review. For example, consider a medical scanwith inference data indicating a particular type of abnormality maypossibly being present with a low probability, where global abnormalityprobabilities is not high enough for the medical scan to be designatedfor triage. If this particular type of abnormality is not particularlysevere, rare, and/or time-sensitive, accidently placing this medicalscan in the normal group may not be particularly detrimental. However,if this particular type of abnormality is particularly severe, rare,and/or time-sensitive, accidently placing this medical scan in thenormal group could be very detrimental. The severity, rarity, and/ortime-sensitivity threshold for triage can be determined based on userinput to interactive interface 8075 and/or can be determinedautomatically by the medical scan triaging system 8002. Additionaltriaging probability thresholds for one or more correspondingprobabilities for one or more particular types of abnormalities can alsobe utilized to dictate whether a severe, rare, and/or time-sensitivetype of finding has a high enough probability of warranting review.Thus, determining whether a medical scan is grouped in the triage groupcan be a function of its severity, rarity, and/or time-sensitivity valuein addition to its global abnormality probability.

In various embodiments, a medical scan triaging system 8002 includes atleast one processor and a memory that stores operational instructionsthat, when executed by the at least one processor, cause the medicalscan triaging system to receive, via a receiver, a plurality of medicalscans for review. Abnormality data can be generated for each of theplurality of medical scans by performing an inference function on imagedata of each of the plurality of medical scans. The inference functioncan utilize a computer vision model trained on a training set of medicalscans, and the abnormality data for the each of the plurality of medicalscans can indicate a global abnormality probability indicating aprobability that an abnormality is present in the each of the pluralityof medical scans. A triage probability threshold can be determined basedon user input to a first client device in response to a prompt presentedvia an interactive interface. The interactive interface can be displayedon a first display device associated with the first client device.

A first subset of the plurality of medical scans can be determined byidentifying ones of the plurality of medical scans with a correspondingglobal abnormality probability that compares favorably to the triageprobability threshold. A second subset of the plurality of medical scanscan be determined by identifying ones of the plurality of medical scanswith a corresponding global abnormality probability that comparesunfavorably to the triage probability threshold. The first subset of theplurality of medical scans can be designated for human review, and thesecond subset of the plurality of medical scans can be designated asnormal, where human review of the second subset of the plurality ofmedical scans is automatically waived. Transmission of the first subsetof the plurality of medical scans to a plurality of client devicesassociated with a plurality of users can be facilitated. The firstsubset of the plurality of medical scans can be displayed to theplurality of users via a plurality of display devices associated withthe plurality of users.

FIG. 20 presents a flowchart illustrating a method for execution by amulti-model medical scan analysis system 6002 that stores executableinstructions that, when executed by at least one processor, cause themulti-model medical scan analysis system 6002 to perform the stepsbelow.

Step 9002 includes receiving, via a receiver, a plurality of medicalscans. Step 9004 includes generating a plurality of training sets fromthe plurality of medical scans. Step 9006 includes generating each of aset of sub-models by performing a training step on a corresponding oneof the plurality of training sets of the plurality of medical scans.Step 9008 includes receiving, via the receiver, a new medical scan. Step9010 includes selecting a subset of the set of sub-models based on thenew medical scan. Step 9012 includes generating a set of abnormalitydata by applying a subset of a set of inference functions on the newmedical scan. The subset of the set of inference functions utilize thesubset of the set of sub-models, and each of the set of abnormality datais generated as output of performing one of the subset of the set ofinference functions on the new medical scan. Step 9014 includesgenerating final abnormality data by performing a final inferencefunction on the set of abnormality data. Step 9016 includes transmittingthe final abnormality data to a client device for display via a displaydevice.

In various embodiments, each of the set of sub-models corresponds to oneof a set of medical scan classification categories. Each of theplurality of training sets are generated to include ones of theplurality of medical scans of a corresponding one of the set of medicalscan classification categories. The subset of the set of sub-models areselected by determining a corresponding subset of the set of medicalscan classification categories that compare favorably to the new medicalscan. In various embodiments, the set of medical scan classificationcategories correspond to a plurality of medical scan modalities, aplurality of medical scan views, and/or a plurality of anatomicalregions.

In various embodiments, the new medical scan is received in a study fora patient that includes a plurality of medical scans. The subset of theset of sub-models is selected based on the plurality of medical scans inthe study, and at least two of the subset of the set of inferencefunctions are performed on different ones of the plurality of medicalscans to generate the set of abnormality data. In various embodiments,the set of abnormality data indicates a plurality of probabilities thatindicate whether an abnormality is present in the different ones of theplurality of medical scans. The final inference function utilizes aBayesian model to generate a final probability that the abnormality ispresent in the patient given the plurality of probabilities of the setof abnormality data.

In various embodiments, a first one of the subset of the set ofsub-models is trained to detect a first type of abnormality, and asecond one of the subset of the set of sub-models is trained to detect asecond type of abnormality. A first one of the set of abnormality datais generated as output of a first one of the subset of the set ofinference functions that corresponds to the first one of the subset ofthe set of sub-models. A second one of the set of abnormality data isgenerated as output of a second one of the subset of the set ofinference functions that corresponds to the first one of the subset ofthe set of sub-models. The first one of the set of abnormality dataindicates a probability that the first type of abnormality is present,and the second one of the set of abnormality data indicates a secondprobability that second type of abnormality is present.

In various embodiments, the set of abnormality data indicates aplurality of probabilities that indicate whether a plurality ofabnormality types are present. The final inference function utilizes aBayesian model to generate final probabilities that each of theplurality of abnormality types are present given the plurality ofprobabilities of the set of abnormality data. In various embodiments,the final inference function utilizes a plurality of known correlationsbetween different types of abnormalities, and the final abnormality datais generated based on a known correlation between the first type ofabnormality and the second type of abnormality.

In various embodiments, a training step is performed on a plurality ofsets of abnormality data to generate a hyper-model. Each of theplurality of sets of abnormality data indicate whether each of theplurality of types of abnormalities are present in a corresponding studyin a training set of studies, where the final inference functionutilizes the hyper-model, and where the plurality of known correlationsare determined as a result of generating the hyper-model.

In various embodiments, the final abnormality data indicates an increasein the probability that the second type of abnormality is present inresponse to the set of abnormality data indicating the probability thatthe first type of abnormality is present compares favorably to adetection threshold and in response to the known correlation between thefirst type of abnormality and the second type of abnormality comparingfavorably to a correlation threshold. In various embodiments, the finalabnormality data indicates a decrease in the probability that the secondtype of abnormality is present in response to the set of abnormalitydata indicating the probability that the first type of abnormality ispresent compares unfavorably to a detection threshold and in response tothe known correlation between the first type of abnormality and thesecond type of abnormality comparing favorably to a correlationthreshold.

In various embodiments, the method further includes partitioning data ofthe plurality of medical scans into at least two partitioned dataportions in accordance with a plurality of partitioning categories. Atleast two of the plurality of training sets are generated to includepartitioned data portions of a corresponding one plurality ofpartitioning categories. The set of sub-models includes at least twosub-models that are each generated by performing the training step onthe partitioned data portions of a corresponding one of the at least twoof the plurality of training sets. Data of the new medical scan ispartitioned into at least two new partitioned data portions. Each of atleast two inference functions corresponding to the at least twosub-models are applied to a corresponding one of the at least two newpartitioned data portions to generate the set of abnormality data.

In various embodiments, the set of abnormality data indicates aplurality of probabilities that indicate whether an abnormality ispresent in different partitioned data portions of the new medical scan.The final inference function utilizes a Bayesian model to generate afinal probability that the abnormality is present in the new medicalscan given the plurality of probabilities of the set of abnormalitydata.

In various embodiments, image data of the medical scans is partitionedby a plurality of anatomical subregion types. The plurality of trainingsets includes a set of anatomical subregion type subsets that eachinclude image data of the plurality of medical scans of a correspondingone of the plurality of anatomical subregion types. The image data ofthe new medical scan is partitioned into a plurality of sub-regions inaccordance with the plurality of anatomical sub-region types. At leasttwo of the subset of the set of inference functions are performed on ofdifferent ones of the plurality of sub-regions of the image data of thenew medical scan.

In various embodiments, the plurality of medical scans include imagedata and text data. Generating the plurality of training sets includesgenerating at least one image data training set that includes image dataof the plurality of medical scans. Generating the plurality of trainingsets further includes generating at least one text data training setthat includes text data of the plurality of medical scans. A first oneof the set of abnormality data is generated by applying a first one ofthe set of inference functions on image data of the new medical scan,where the first one of the set of inference functions utilizes acomputer vision model generated by performing the training step on theimage data training set. A second one of the set of abnormality data isgenerated by applying a second one of the set of inference functions ontext data of the new medical scan, where the second one of the set ofinference functions utilizes a natural language model generated byperforming the training step on the text data training set.

In various embodiments, the medical scans are partitioned by sequence.The plurality of training sets includes a set of sequence type subsetsthat each include sequences of the plurality medical scans of acorresponding one of a plurality of sequence types. The new medical scanincludes a plurality of sequences, and at least two of the subset of theset of inference functions are performed on different ones of theplurality of sequences.

In various embodiments, a first one of the set of abnormality data isgenerated as output of performing a first one of the subset of the setof inference functions on a first one of the plurality of sequences ofthe new medical scan that corresponds to a first type of sequence. Thefirst one of the subset of the set of inference functions utilizes afirst one of the subset of the set of sub-models that was trained on thefirst type of sequence. A second one of the set of abnormality data isgenerated as output of performing a second one of the subset of the setof inference functions on a second one of the plurality of sequences ofthe new medical scan that corresponds to a second type of sequence. Thesecond one of the subset of the set of inference functions utilizes asecond one of the subset of the set of sub-models that was trained onthe second type of sequence. The first one of the set of abnormalitydata indicates a first probability that an abnormality is present in thefirst one of the plurality of sequences. The second one of the set ofabnormality data indicates a second probability that the abnormality ispresent in the second one of the plurality of sequences. The finalabnormality data indicates a final probability that the abnormality ispresent in the new medical scan, where the final probability isgenerated based on the first probability and the second probability.

In various embodiments, a first one of the set of sub-model is selectedfor performance on the new medical scan, based on the new medical scan.First abnormality data is generated by applying a first one of the setof inference functions on the new medical scan, where the first one ofthe set of inference functions utilizes the first one of the set ofsub-models. A second one of the set of sub-models is selected forperformance on the new medical scan based on the first abnormality data.Second abnormality data is generated by applying a second one of the setof inference functions on the new medical scan. The second one of theset of inference functions utilizes the second one of the set ofsub-models. The final abnormality data is generated based on the secondabnormality data.

In various embodiments, a sub-region of image data of the new medicalscan is selected based on the first abnormality data. One of the set ofsub-models that is trained on one of a plurality of anatomicalsub-region types that corresponds to the sub-region is selected as thesecond one of the set of sub-models, where the second one of the set ofinference functions is performed only on the sub-region of the imagedata of the new medical scan.

FIG. 21 presents a flowchart illustrating a method for execution by amulti-model medical scan analysis system 6002 that stores executableinstructions that, when executed by at least one processor, cause themulti-model medical scan analysis system 6002 to perform the stepsbelow. Some or all of the steps of FIG. 21 can be performed by themulti-model medical scan analysis system 6002 instead of or in additionto some or all of the steps of FIG. 20.

Step 9102 includes receiving, via a receiver, a plurality of medicalscans. Each of the plurality of medical scans includes correspondinglabeling data indicating whether each of a plurality of abnormalitytypes are present in the each of the plurality of medical scans. Invarious embodiments, the plurality of medical scans correspond to chestx-rays. Step 9104 includes generating a generic model by performing atraining step on image data of the plurality of medical scans and thecorresponding labeling data. Step 9106 includes generating a pluralityof fine-tuned models, where each of the plurality of fine-tuned modelsis generated by performing a fine-tuning step on the generic model, andwhere each of the plurality of fine-tuned models corresponds to one ofthe plurality of abnormality types. Step 9108 includes receiving, via areceiver, a new medical scan. Step 9110 includes generating abnormalitydetection data for the new medical scan by performing a genericinference function on image data of the new medical scan. The genericinference function utilizes the generic model, and the abnormalitydetection data indicates a plurality of probability values that eachindicate a probability that a corresponding one of the plurality ofabnormality types is present in the new medical scan. Step 9112 includesdetermining a first one of the plurality of abnormality types that isdetected in the new medical scan by determining a corresponding one ofthe plurality of probability values compares favorably to a detectionthreshold. Step 9114 includes selecting one of the plurality offine-tuned models that corresponds to the first one of the plurality ofabnormality types. Step 9116 includes generating additional abnormalitydata for the new medical scan by performing a fine-tuned inferencefunction on the image data of the new medical scan, where the fine-tunedinference function utilizes the one of the plurality of fine-tunedmodels. Step 9118 includes transmitting the additional abnormality datato a client device for display via a display device. The abnormality canalso be transmitted to the client device for display via the displaydevice.

In various embodiments, the additional abnormality data includes a newprobability value indicating whether the first one of the plurality ofabnormality types is present. In various embodiments, the additionalabnormality data indicates an increased confidence that first one of theplurality of abnormality types is present in the new medical scan inresponse to the new probability value indicating a higher probabilityvalue than the corresponding one of the plurality of probability valuesof the abnormality detection data. In various embodiments, theadditional abnormality data indicates that the first one of theplurality of abnormality types is not present in the new medical scan inresponse to the new probability value comparing unfavorably to thedetection threshold.

In various embodiments, performing the fine-tuning step includesmodifying a plurality of pre-tuned weights of the generic model. Invarious embodiments, performing the fine-tuning step includes performinga training step on a subset of the plurality of medical scans that havecorresponding labeling data indicating the first one of the plurality ofabnormality types is present in the each of the subset of the pluralityof medical scans. Performing the training step includes utilizing a setof initial weights corresponding to the plurality of pre-tuned weightsof the generic model.

In various embodiments, performing the fine-tuning step to generate oneof the plurality of fine-tuned models corresponding to the first one ofthe plurality of abnormality types includes utilizing additionallabeling data of a subset of the plurality of medical scans. Each of thesubset of the plurality of medical scans have corresponding labelingdata indicating the first one of the plurality of abnormality types ispresent in the each of the subset of the plurality of medical scans. Theadditional labeling data indicates characterization labelscharacterizing the first one of the plurality of abnormality types inthe each of the subset of the plurality of medical scans. The additionalabnormality data indicates output data for the characterization labels.

In various embodiments, the output data indicates a plurality ofprobability values indicating whether each of a plurality ofcharacterization types characterize the first one of the plurality ofabnormality types. In various embodiments, the output data indicates asubset of the plurality of characterization types determined tocharacterize the first one of the plurality of abnormality types inresponse to corresponding ones of the plurality of probability valuescomparing favorably to a characterization threshold.

In various embodiments, a second one of the plurality of abnormalitytypes is determined by determining a second corresponding one of theplurality of probability values compares favorably to the detectionthreshold. A second one of the plurality of fine-tuned models thatcorresponds to the second one of the plurality of abnormality types isselected. Second additional abnormality data is generated for the newmedical scan by performing a second fine-tuned inference function on theimage data of the new medical scan. The second fine-tuned inferencefunction utilizes the second one of the plurality of fine-tuned models.The second additional abnormality data is transmitted to a client devicefor display via a display device in conjunction with the additionalabnormality data. In various embodiments, the abnormality detection dataindicates region of interest data indicating a location of the first oneof the plurality of abnormality types in the image data of the newmedical scan. A sub-region of the image data of the new medical scan isselected based on the region of interest data, and the fine-tunedinference function is performed only on the sub-region of the image dataof the new medical scan.

In various embodiments, the method includes determining a subset of theplurality of medical scans that have corresponding labeling dataindicating the first one of the plurality of abnormality types ispresent in the each of the subset of the plurality of medical scans. Aplurality of sub-regions are selected by selecting a sub-region of theimage data of each of the subset of the plurality of medical scans basedon region interest data of the corresponding labeling data indicating alocation of the first one of the plurality of abnormality types in theimage data of the each of the subset of the plurality of medical scans.A training step is performed on the plurality of sub-regions to generatethe one of the plurality of fine-tuned models.

FIG. 22A presents a flowchart illustrating methods for execution by amulti-model medical scan analysis system 6002 implemented as alocation-based medical scan analysis system 6004 that stores executableinstructions that, when executed by at least one processor, cause thelocation-based medical scan analysis system 6004 to perform the stepsbelow.

Step 9202 includes receiving, via a receiver, a plurality of medicalscans. Step 9204 includes generating a generic model by performing atraining step on image data of the plurality of medical scans. Step 9206includes generating a plurality of location-based subsets of theplurality of medical scans. Each of the plurality of location-basedsubsets is generated by including ones of the plurality of medical scanswith originating locations that compare favorably to location groupingcriteria for the each of the plurality of location-based subsets. Step9208 includes generating a plurality of location-based models, whereeach of the plurality of location-based models is generated byperforming a fine-tuning step on the generic model, utilizing acorresponding one of the plurality of location-based subsets.

Inference data is generated for a new medical scan by performing aninference function that utilizes one of the plurality of location-basedmodels on the new medical scan. An originating location associated withthe new medical scan compares favorably to the location groupingcriteria for one of the plurality of location-based subsets utilized togenerate the one of the plurality of location-based models. Theinference data is transmitted to a client device for display via adisplay device.

As illustrated in FIG. 22A, the method can further include step 9210,which includes receiving, via the receiver, the new medical scan. Themethod can further include step 9212, which includes selecting the oneof the plurality of location-based models based on determining thelocation grouping criteria for the one of the plurality oflocation-based subsets utilized to generate the one of the plurality oflocation-based models compares favorably to the originating location ofthe new medical scan. The method can further include step 9214, whichincludes performing the inference function that utilizes the one of theplurality of location-based models on the new medical scan to generatethe inference data is generated for a new medical scan. The method canfurther include step 9216, which includes transmitting the inferencedata to the client device for display via the display device. In variousembodiments, the method can further include determining the originatinglocation of the new medical scan based on determining a location of atransmitting entity, where the new medical scan was transmitted to thelocation-based medical scan analysis system by the transmitting entity.

Alternatively or in addition, the method can further includetransmitting the model data for each of the plurality of location-basedmodels to at least one of a plurality of local systems, where each ofthe plurality of local systems is associated with a location thatcompares favorably to the location grouping criteria for one of theplurality of location-based subsets utilized to generate the one of theplurality of location-based models associated with the model data. Invarious embodiments, at least one of the plurality of local systemsutilizes a medical picture archive integration system. In variousembodiments, some or all of the plurality of local systems includes atleast one second processor and a second memory that stores secondexecutable instructions that, when executed by the at least one secondprocessor, causes a corresponding one of the plurality of local systemsto perform the steps illustrated in FIG. 22B.

Step 9218 includes receiving model data for one of the plurality oflocation-based models from the location-based medical scan analysissystem. Step 9220 includes receiving the new medical scan via a receiverof the one of the plurality of local systems. Step 9222 includesperforming the inference function that utilizes the one of the pluralityof location-based models on the new medical scan to generate theinference data. Step 9224 includes transmitting the inference data tothe client device for display via the display device.

In various embodiments, the location grouping criteria for the pluralityof location-based subsets indicates geographic region criteria. Some orall of the plurality of location-based subsets include medical scansoriginating from each of a plurality of different geographic regions. Invarious embodiments, the location grouping criteria for the plurality oflocation-based subsets indicates hospital criteria. Some or all of theplurality of location-based subsets include medical scans originatingfrom each of a plurality of different hospitals. In various embodiments,the location grouping criteria for the plurality of location-basedsubsets indicates hospital setting criteria. Some or all of theplurality of location-based subsets include medical scans originatingfrom each of a plurality of different hospital settings. The pluralityof different hospital settings include an in-patient setting, anout-patient setting, an emergency care setting, and/or any other typesof hospital settings.

In various embodiments, a plurality of location groupings that optimizeaccuracy of the plurality of location-based models are automaticallydetermined, where the plurality of location-based subsets are determinedin accordance with the plurality of location groupings. In variousembodiments, at least one location-based trend in the image data of theplurality of medical scans is automatically determined. At least one ofthe plurality of location groupings is determined based on a trendlocation associated with the location-based trend.

In various embodiments, an updated generic model is generated byperforming a retraining step on the generic model. The retraining steputilizes the new medical scan and the inference data. A plurality ofupdated location-based models are generated, where each of the pluralityof updated location-based models is generated by performing afine-tuning step on the updated generic model. Second inference data isgenerated for a second new medical scan by performing an inferencefunction that utilizes one of the plurality of updated location-basedmodels on the second new medical scan. A second originating locationassociated with the second new medical scan compares favorably to thelocation grouping criteria for one of the plurality of location-basedsubsets utilized to generate the one of the plurality of location-basedmodels. The second inference data is transmitted to the client devicefor display via the display device. In various embodiments, theretraining step further utilizes model parameters of at least one of theplurality of location-based models. In various embodiments, generatingthe plurality of updated location-based models includes changing thelocation grouping criteria for at least one of the plurality oflocation-based subsets.

FIG. 23 presents a flowchart illustrating a method for execution by amodel-assisted annotating system 7002 that stores executableinstructions that, when executed by at least one processor, cause themodel-assisted annotating system 7002 to perform the steps below.

Step 9302 includes receiving, via a receiver, a first set of medicalscans for review. Step 9304 includes transmitting, via a transmitter,the first set of medical scans to a set of client devices associatedwith a set of users. Each of the first set of medical scans is displayedto one of the set of users via an interactive interface displayed by adisplay device associated with a corresponding one of the set of clientdevices. Step 9306 includes receiving a first set of annotation datafrom the set of client devices. Each of the first set of annotation datais generated by one of the set of client devices in response to a promptvia the interactive interface displayed by the display device to provideannotation data for one of the first set of medical scans displayed bythe one of the set of client devices. Step 9308 includes performing afirst training step to train a computer vision model by utilizing imagedata of the first set of medical scans as input labels and by utilizingthe first set of annotation data as output labels. Step 9310 includesreceiving, via the receiver, a second set of medical scans for review.Step 9312 includes generating a second set of annotation data byperforming an inference function on image data of the second set ofmedical scans. The inference function utilizes the computer visionmodel, and each of the second set of annotation data is generated asoutput of performing the inference function on a corresponding one ofthe second set of medical scans. Step 9314 includes transmitting, viathe transmitter, the second set of medical scans and the second set ofannotation data to the set of client devices. Each of the second set ofmedical scans is displayed to one of the set of users in conjunctionwith a corresponding one of the second set of annotation data via theinteractive interface. Step 9316 includes receiving a set of additionalannotation data from the set of client devices. Each of the set ofadditional annotation data is generated by one of the set of clientdevices in response to a prompt via the interactive interface displayedby the display device to provide additional annotations to thecorresponding one of the second set of annotation data displayed inconjunction with one of the second set of medical scans by the one ofthe set of client devices. Step 9318 includes performing a secondtraining step to generate an updated computer vision model by utilizingthe set of additional annotation data to update the computer visionmodel. Step 9320 includes receiving, via the receiver, a third set ofmedical scans for review. Step 9322 includes generating a third set ofannotation data by performing an updated inference function on imagedata of the third set of medical scans. The updated inference functionutilizes the updated computer vision model, and each of the third set ofannotation data is generated as output of performing the updatedinference function on a corresponding one of the third set of medicalscans. Step 9324 includes transmitting, via the transmitter, the thirdset of medical scans and the third set of annotation data to the set ofclient devices. Each of the third set of medical scans is displayed toone of the set of users in conjunction with a corresponding one of thethird set of annotation data via the interactive interface.

In various embodiments, the set of additional annotation data includes aset of corrected annotation data. Each of the set of correctedannotation data is generated by one of the set of client devices inresponse to a prompt via the interactive interface displayed by thedisplay device to provide corrections to the corresponding one of thesecond set of annotation data displayed in conjunction with one of thesecond set of medical scans. The second training step utilizes imagedata of the second set of medical scans as input labels and utilizes theset of corrected annotation data as output labels.

In various embodiments, performing the first training step includesutilizing image data as input labels and further includes utilizing aglobal binary abnormality label for each of the first set of annotationdata as output labels. The second set of annotation data includes aglobal binary abnormality label for each of the second set of medicalscans. The set of additional annotation data includes a set ofabnormality characterization data, where each of the set of abnormalitycharacterization data is generated by one of the set of client devicesin response to a prompt via the interactive interface displayed by thedisplay device to characterize an abnormality indicated in thecorresponding one of the second set of annotation data displayed inconjunction with one of the second set of medical scans. The secondtraining step utilizes image data of the second set of medical scans asinput labels and utilizes the set of abnormality characterization dataas output labels.

In various embodiments, the third set of annotation data includesabnormality characterization data generated as output of the updatedinference function. The abnormality characterization data of each of thethird set of annotation data is displayed to one of the set of users inconjunction with a corresponding one of the third set of annotation datavia the interactive interface.

In various embodiments, the second set of annotation data includesregion of interest data generated as output of the inference function,where the region of interest data of one of the second set of annotationdata is utilized to indicate a location of an abnormality indicated inthe one of the second set of annotation data in the display of acorresponding one of the second set of medical scans via the interactiveinterface. In various embodiments, the interactive interface displayssegmentation data outlining the abnormality overlaying the image data ofthe corresponding one of the second set of medical scans.

In various embodiments, inference trend data is generated byautomatically evaluating the third set of annotation data to identify atleast one trend. A third training step is performed to generate a secondupdated computer vision model based on the inference trend data. Invarious embodiments, each of the third set of annotation data includes aconfusion matrix indicating a plurality of probability values for aplurality of labels. Generating the inference trend data includesidentifying at least one weak label of the plurality of labels based ondetermining corresponding probability values in confusion matrices ofthe third set of annotation data compare unfavorably to a confidencethreshold.

In various embodiments, additional training data criteria correspondingto the at least one weak label is determined. A request for additionaltraining data that indicates the additional training data criteria istransmitted, via the transmitter. A fourth set of medical scans isreceived, via the receiver, in response to the request for additionaltraining data. Performing the third training step includes utilizing thefourth set of medical scans as input labels and further includesutilizing a fourth set of annotation data for the fourth set of medicalscans as output labels.

In various embodiments, the additional training data criteria indicatesat least one type of abnormality. In various embodiments, the third setof medical scans correspond to a plurality of scan types. Generating theinference trend data further includes grouping the third set of medicalscans by the plurality of scan types into a plurality of groupings andseparately evaluating the third set of annotation data for each of theplurality of groupings. The at least one weak label is identified forone of the plurality of groupings, and the additional training datacriteria indicates one of the plurality of scan types corresponding tothe one of the plurality of groupings.

In various embodiments, the fourth set of medical scans is transmitted,via the transmitter, to the set of client devices. Each of the fourthset of medical scans is displayed to one of the set of users via theinteractive interface displayed by a display device associated with acorresponding one of the set of client devices. The fourth set ofannotation data is received from the set of client devices. Each of thefourth set of annotation data is generated by one of the set of clientdevices in response to a prompt via the interactive interface displayedby the display device to provide annotation data for one of the fourthset of medical scans displayed by the one of the set of client devices.

In various embodiments, each of the third set of annotation dataincludes a confusion matrix indicating a plurality of probability valuesfor a plurality of labels. A plurality of desired label dependency rulesare determined. Each of the plurality of desired label dependency rulesindicates at least two of the plurality of labels and each of theplurality of desired label dependency rules further indicates a desiredrelationship between a corresponding at least two of the plurality ofprobability values for the at least two of the plurality of labels.Generating the inference trend data includes determining probabilityvalues in confusion matrices of the third set of annotation data compareunfavorably to at least one of the plurality of desired label dependencyrules. In various embodiments, the plurality of labels includes a globalabnormality label. One of the plurality of probability values for theglobal abnormality label indicates a probability that any abnormality ispresent. The plurality of labels further includes a set of abnormalityclassification labels. Each one of a set of the plurality of probabilityvalues for the set of abnormality classification labels indicates aprobability that an abnormality of a classification indicated by acorresponding one of the set of abnormality classification labels ispresent.

A first one of the set of the plurality of desired label dependencyrules indicates that if at least one of the set of the plurality ofprobability values for the set of abnormality classification labelsexceeds a detection probability threshold, then the one of the pluralityof probability values for the global abnormality label should alsoexceed the detection probability threshold. A second one of set of theplurality of desired label dependency rules indicates that if the one ofthe plurality of probability values for the global abnormality labelexceeds the detection probability threshold, then at least one of theset of the plurality of probability values for the set of abnormalityclassification labels should also exceed the detection probabilitythreshold. A third one of the set of the plurality of desired labeldependency rules indicates that if none of the set of the plurality ofprobability values for the set of abnormality classification labelsexceed the detection probability threshold, then the one of theplurality of probability values for the global abnormality label shouldnot exceed the detection probability threshold. A fourth one of the setof the plurality of desired label dependency rules indicates that if theone of the plurality of probability values for the global abnormalitylabel does not exceed the detection probability threshold, then none ofthe set of the plurality of probability values for the set ofabnormality classification labels should exceed the detectionprobability threshold.

FIG. 24 presents a flowchart illustrating a method for execution by amedical scan triaging system 8002 that stores executable instructionsthat, when executed by at least one processor, cause the medical scantriaging system 8002 to perform the steps below.

Step 9402 includes receiving, via a receiver, a plurality of medicalscans for review. Step 9404 includes generating abnormality data foreach of the plurality of medical scans by performing an inferencefunction on image data of each of the plurality of medical scans. Theinference function utilizes a computer vision model trained on atraining set of medical scans, and the abnormality data for the each ofthe plurality of medical scans indicates a global abnormalityprobability indicating a probability that an abnormality is present inthe each of the plurality of medical scans. Step 9406 includesdetermining a triage probability threshold based on user input to afirst client device in response to a prompt presented via an interactiveinterface. The interactive interface is displayed on a first displaydevice associated with the first client device. Step 9408 includesdetermining a first subset of the plurality of medical scans byidentifying ones of the plurality of medical scans with a correspondingglobal abnormality probability that compares favorably to the triageprobability threshold. Step 9410 includes determining a second subset ofthe plurality of medical scans by identifying ones of the plurality ofmedical scans with a corresponding global abnormality probability thatcompares unfavorably to the triage probability threshold. Step 9412includes designating the first subset of the plurality of medical scansfor human review. Step 9414 includes designating the second subset ofthe plurality of medical scans as normal, where human review of thesecond subset of the plurality of medical scans is automatically waived.Step 9416 includes facilitating transmission of the first subset of theplurality of medical scans to a plurality of client devices associatedwith a plurality of users. The first subset of the plurality of medicalscans are displayed to the plurality of users via a plurality of displaydevices associated with the plurality of users.

In various embodiments, an updated triage probability threshold isdetermined based on user input to the first client device in response toa second prompt presented via an interactive interface. An updated firstsubset of the plurality of medical scans is determined by identifyingones of the plurality of medical scans with a corresponding globalprobability that compares favorably to the updated triage probabilitythreshold. An updated second subset of the plurality of medical scans isdetermined by identifying ones of the plurality of medical scans with acorresponding global probability that compares unfavorably to theupdated triage probability threshold. The updated first subset of theplurality of medical scans is designated for human review. The updatedsecond subset of the plurality of medical scans is designated as normal,where human review of the updated second subset of the plurality ofmedical scans is automatically waived.

In various embodiments, the updated triage probability threshold isdetermined based on second user input to the first client device inresponse to a second prompt presented via the interactive interface. Invarious embodiments, at least one of the plurality of medical scans hasa global abnormality probability that greater than a first probabilityvalue indicated by the triage probability threshold and is less than asecond probability value indicated by the updated triage probabilitythreshold. The at least one of the plurality of medical scans in thefirst subset is removed to generate the updated first subset and isadded to the second subset to generate the updated second subset, andwhere a notification is transmitted to at least one of the plurality ofclient devices indicating that the at least one of the plurality ofmedical scans no longer needs review.

In various embodiments, at least one of the plurality of medical scanshas a global abnormality probability that less than a first probabilityvalue indicated by the triage probability threshold and is greater thana second probability value indicated by the updated triage probabilitythreshold, where at least one of the plurality of medical scans in thesecond subset is removed to generate the updated second subset and isadded to the first subset to generate the updated first subset, andwhere transmission of the at least one of the plurality of medical scansis transmitted to at least one of the plurality of client devices fordisplay to at least one of the plurality of users.

In various embodiments, an expected average review time is determinedbased on a number of medical scans in the first subset of the pluralityof medical scans. The method further includes automatically determiningto update the triage probability threshold in response to determiningthe expected average review time compares favorably to a low review timethreshold.

In various embodiments, the abnormality data for at least one of theplurality of medical scans further includes a time-sensitivity valueindicating a time-sensitivity of a detected abnormality in theabnormality data. A triage time-sensitivity threshold is determinedbased on user input to the first client device in response to a secondprompt presented via the interactive interface. A third subset of theplurality of medical scans is determined by identifying ones of thesecond subset of the plurality of medical scans with a correspondingtime-sensitivity value that compares favorably to the triagetime-sensitivity threshold. The third subset of the plurality of medicalscans is designated for human review. Transmission of the third subsetof the plurality of medical scans to the plurality of client devicesassociated with the plurality of users is facilitated. The third subsetof the plurality of medical scans are displayed to the plurality ofusers via a plurality of display devices associated with the pluralityof users.

In various embodiments, review assignment data is generated by assigningeach of the first subset of the plurality of medical scans to one of theplurality of users for review. Transmission of the first subset of theplurality of medical scans to the plurality of client devices isfacilitated in accordance with the review assignment data. In variousembodiments, the abnormality data for each of the plurality of medicalscans further includes a time-sensitivity value indicating atime-sensitivity of a detected abnormality in the abnormality data.Priority data is generated by determining a priority value for each thefirst subset of the plurality of medical scans in as a function of theglobal abnormality probability and the time-sensitivity value. Thereview assignment data is generated based on the priority data.

In various embodiments, one of the plurality of users is assigned atleast two of the first subset of the plurality of medical scans in thereview assignment data. Review ordering data is generated for the atleast two of the first subset of the plurality of medical scans based onthe priority data. The at least two of the first subset of the pluralityof medical scans are displayed to the one of the plurality of users inan ordering dictated by the review ordering data.

In various embodiments, the review assignment data is generated byassigning each of the first subset of the plurality of medical scans toone of the plurality of users with qualification data that correspondsto an anatomical region of the each of the first subset of the pluralityof medical scans. In various embodiments, the abnormality data for eachof the plurality of medical scans further includes abnormalityclassification data characterizing the abnormality. The reviewassignment data is generated by assigning each of the first subset ofthe plurality of medical scans to one of the plurality of users withqualification data that corresponds to the abnormality classificationdata of the each of the first subset of the plurality of medical scans.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “configured to”, “operably coupled to”, “coupled to”, and/or“coupling” includes direct coupling between items and/or indirectcoupling between items via an intervening item (e.g., an item includes,but is not limited to, a component, an element, a circuit, and/or amodule) where, for an example of indirect coupling, the intervening itemdoes not modify the information of a signal but may adjust its currentlevel, voltage level, and/or power level. As may further be used herein,inferred coupling (i.e., where one element is coupled to another elementby inference) includes direct and indirect coupling between two items inthe same manner as “coupled to”. As may even further be used herein, theterm “configured to”, “operable to”, “coupled to”, or “operably coupledto” indicates that an item includes one or more of power connections,input(s), output(s), etc., to perform, when activated, one or more itscorresponding functions and may further include inferred coupling to oneor more other items. As may further be used herein, the term “associatedwith”, includes direct and/or indirect coupling of separate items and/orone item being embedded within another item. As may still further beused herein, the term “automatically” refers to an action causeddirectly by a processor of a computer network in response to atriggering event and particularly without human interaction.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing device” and/or “processing unit” maybe a single processing device or a plurality of processing devices. Sucha processing device may be a microprocessor, micro-controller, digitalsignal processor, graphics processing unit, microcomputer, centralprocessing unit, field programmable gate array, programmable logicdevice, state machine, logic circuitry, analog circuitry, digitalcircuitry, and/or any device that manipulates signals (analog and/ordigital) based on hard coding of the circuitry and/or operationalinstructions. The processing module, module, processing circuit, and/orprocessing unit may be, or further include, memory and/or an integratedmemory element, which may be a single memory device, a plurality ofmemory devices, and/or embedded circuitry of another processing module,module, processing circuit, and/or processing unit. Such a memory devicemay be a read-only memory, random access memory, volatile memory,non-volatile memory, static memory, dynamic memory, flash memory, cachememory, and/or any device that stores digital information. Note that ifthe processing module, module, processing circuit, and/or processingunit includes more than one processing device, the processing devicesmay be centrally located (e.g., directly coupled together via a wiredand/or wireless bus structure) or may be distributedly located (e.g.,cloud computing via indirect coupling via a local area network and/or awide area network). Further note that if the processing module, module,processing circuit, and/or processing unit implements one or more of itsfunctions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, and/or processing unit executes, hard coded and/oroperational instructions corresponding to at least some of the stepsand/or functions illustrated in one or more of the Figures and/ordescribed herein. Such a memory device or memory element can be includedin an article of manufacture. While the processing module, module,processing circuit, and/or processing unit device may be a generalpurpose computing device, the execution of the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit configures such a general purposecomputing device as a special purpose computing device to implement thecorresponding steps and/or functions illustrated in one or more of theFigures and/or described herein. In particular, the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit implement acts and algorithms performedby the processing module, module, processing circuit, and/or processingunit. Such acts and algorithms can be identified by name, can beillustrated via flowchart and/or described in words.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with other routines. In this context, “start” indicates thebeginning of the first step presented and may be preceded by otheractivities not specifically shown. Further, the “continue” indicationreflects that the steps presented may be performed multiple times and/ormay be succeeded by other activities not specifically shown. Further,while a flow diagram indicates a particular ordering of steps, otherorderings are likewise possible provided that the principles ofcausality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

The term “system” is used in the description of one or more of theembodiments. A system implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A system may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a system may containone or more sub-system, each of which may be one or more systems.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. The memory device may be in a form a solidstate memory, a hard drive memory, cloud memory, thumb drive, servermemory, computing device memory, and/or other physical medium forstoring digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A model-assisted annotating system, comprising:at least one processor; and a memory that stores operationalinstructions that, when executed by the at least one processor, causethe model-assisted annotating system to: receive different ones of afirst set of annotation data, wherein the first set of annotation datacorresponds to a broad type of annotation data output, wherein each ofthe first set of annotation data is generated by a corresponding one ofa set of client devices based on a broad prompt presented via thecorresponding one of the set of client devices to provide annotationdata for one of a first set of medical scans displayed by thecorresponding one of the set of client devices; perform a first trainingstep to train a computer vision model operable to generate the broadtype of annotation data output by utilizing image data of the first setof medical scans and the first set of annotation data; generate a secondset of annotation data corresponding to the broad type of annotationdata output by performing an inference function on image data of asecond set of medical scans, wherein the inference function utilizes thecomputer vision model, and wherein each of the second set of annotationdata is generated as output of performing the inference function on acorresponding one of the second set of medical scans; receive differentones of a set of additional annotation data, wherein the set ofadditional annotation data corresponds to a specified type of annotationdata output that further specifies the broad type of annotation dataoutput, wherein each of the set of additional annotation data isgenerated by a corresponding one of the set of client devices based on aspecified prompt presented via the corresponding one of the set ofclient devices to provide additional annotations to further specify thecorresponding one of the second set of annotation data displayed inconjunction with a corresponding one of the second set of medical scansby the corresponding one of the set of client devices; perform a secondtraining step to generate an updated computer vision model operable togenerate the specified type of annotation data output by utilizing theset of additional annotation data to update the computer vision model;generate a third set of annotation data corresponding to the specifiedtype of annotation data output by performing an updated inferencefunction on image data of a third set of medical scans, wherein theupdated inference function utilizes the updated computer vision model,and wherein each of the third set of annotation data is generated asoutput of performing the updated inference function on a correspondingone of the third set of medical scans; and transmit the third set ofmedical scans and the third set of annotation data for display.
 2. Themodel-assisted annotating system of claim 1, wherein the set ofadditional annotation data includes a set of corrected annotation data,wherein each of the set of corrected annotation data is generated by oneof the set of client devices in response to a prompt to providecorrections to the corresponding one of the second set of annotationdata displayed in conjunction with one of the second set of medicalscans, and wherein the second training step utilizes image data of thesecond set of medical scans as input labels and utilizes the set ofcorrected annotation data as output labels.
 3. The model-assistedannotating system of claim 1, wherein at least one of: at least two thefirst set of annotation data is generated by a same one of the set ofclient devices; at least two the second set of annotation data isgenerated by the same one of the set of client devices; or at least oneof the first set of annotation data and at least one of the second setof annotation data is generated by the same one of the set of clientdevices.
 4. The model-assisted annotating system of claim 1, wherein thebroad type of annotation data output corresponds to global binaryabnormality data, wherein performing the first training step includesutilizing image data as input labels and further includes utilizing aglobal binary abnormality label for each of the first set of annotationdata as output labels, wherein the second set of annotation dataincludes a global binary abnormality label for each of the second set ofmedical scans, wherein the specified type of annotation data outputcorresponds to abnormality characterization data, wherein the set ofadditional annotation data includes a set of abnormalitycharacterization data, wherein each of the set of abnormalitycharacterization data is generated by one of the set of client devicesbased on the specified prompt to characterize an abnormality indicatedin the corresponding one of the second set of annotation data displayedin conjunction with one of the second set of medical scans, and whereinthe second training step utilizes image data of the second set ofmedical scans as input labels and utilizes the set of abnormalitycharacterization data as output labels; wherein the third set ofannotation data includes abnormality characterization data generated asoutput of the updated inference function, and wherein the abnormalitycharacterization data of each of the third set of annotation data isdisplayed in conjunction with a corresponding one of the third set ofannotation data.
 5. The model-assisted annotation system of claim 1,wherein the first set of annotation data is generated by first subset ofthe set of client devices, wherein the second set of annotation data isgenerated by second subset of the set of client devices, and wherein thefirst subset and the second subset are mutually exclusive with respectto the set of client devices.
 6. The model-assisted annotation system ofclaim 1, wherein the broad type of annotation data output corresponds toa first node level of a prompt decision tree, wherein the specified typeof annotation data output corresponds to a second node level branchingfrom the first node level of the prompt decision tree, wherein each ofthe first set of annotation data is generated based on the broad promptcorresponding to the first node level, wherein each of the set ofadditional annotation data is generated based on the specified promptcorresponding to the second node level, and wherein the specified promptis automatically selected for the each of the second set of medicalscans based on applying corresponding ones of the second set ofannotation data to the prompt decision tree.
 7. The model-assistedannotation system of claim 1, wherein the operational instructions, whenexecuted by the at least one processor, further cause the model-assistedannotating system to: generate inference trend data by automaticallyevaluating the third set of annotation data to identify at least onetrend; and perform a third training step to generate a second updatedcomputer vision model based on the inference trend data.
 8. Themodel-assisted annotation system of claim 7, wherein the specified typeof annotation data output includes confusion matrix data, wherein eachof the third set of annotation data includes a confusion matrixindicating a plurality of probability values for a plurality of labels,and wherein generating the inference trend data includes identifying atleast one weak label of the plurality of labels based on determiningcorresponding probability values in confusion matrices of the third setof annotation data compare unfavorably to a confidence threshold.
 9. Themodel-assisted annotation system of claim 8, wherein the operationalinstructions, when executed by the at least one processor, further causethe model-assisted annotating system to: determine additional trainingdata criteria corresponding to the at least one weak label; transmit arequest for additional training data that indicates the additionaltraining data criteria; and receive a fourth set of medical scans inresponse to the request for additional training data; wherein performingthe third training step includes utilizing the fourth set of medicalscans as input labels and further includes utilizing a fourth set ofannotation data for the fourth set of medical scans as output labels.10. The model-assisted annotation system of claim 9, wherein theadditional training data criteria indicates at least one type ofabnormality.
 11. The model-assisted annotation system of claim 9,wherein the third set of medical scans correspond to a plurality of scantypes, wherein generating the inference trend data further includesgrouping the third set of medical scans by the plurality of scan typesinto a plurality of groupings and separately evaluating the third set ofannotation data for each of the plurality of groupings, wherein the atleast one weak label is identified for one of the plurality ofgroupings, and wherein the additional training data criteria indicatesone of the plurality of scan types corresponding to the one of theplurality of groupings.
 12. The model-assisted annotation system ofclaim 9, wherein the operational instructions, when executed by the atleast one processor, further cause the model-assisted annotating systemto: receive the fourth set of annotation data, wherein each of thefourth set of annotation data is generated by one of the set of clientdevices in response to a prompt via to provide annotation data for oneof the fourth set of medical scans displayed by the one of the set ofclient devices.
 13. The model-assisted annotation system of claim 7,wherein each of the third set of annotation data includes a confusionmatrix indicating a plurality of probability values for a plurality oflabels, and wherein the operational instructions, when executed by theat least one processor, further cause the model-assisted annotatingsystem to: determine a plurality of desired label dependency rules,wherein each of the plurality of desired label dependency rulesindicates at least two of the plurality of labels and wherein each ofthe plurality of desired label dependency rules further indicates adesired relationship between a corresponding at least two of theplurality of probability values for the at least two of the plurality oflabels, wherein generating the inference trend data includes determiningprobability values in confusion matrices of the third set of annotationdata compare unfavorably to at least one of the plurality of desiredlabel dependency rules.
 14. The model-assisted annotation system ofclaim 13, wherein the plurality of labels includes a global abnormalitylabel, wherein one of the plurality of probability values for the globalabnormality label indicates a probability that any abnormality ispresent, wherein the plurality of labels further includes a set ofabnormality classification labels, wherein each one of a set of theplurality of probability values for the set of abnormalityclassification labels indicates a probability that an abnormality of aclassification indicated by a corresponding one of the set ofabnormality classification labels is present; wherein a first one of theset of the plurality of desired label dependency rules indicates that ifat least one of the set of the plurality of probability values for theset of abnormality classification labels exceeds a detection probabilitythreshold, then the one of the plurality of probability values for theglobal abnormality label should also exceed the detection probabilitythreshold; wherein a second one of set of the plurality of desired labeldependency rules indicates that if the one of the plurality ofprobability values for the global abnormality label exceeds thedetection probability threshold, then at least one of the set of theplurality of probability values for the set of abnormalityclassification labels should also exceed the detection probabilitythreshold; wherein a third one of the set of the plurality of desiredlabel dependency rules indicates that if none of the set of theplurality of probability values for the set of abnormalityclassification labels exceed the detection probability threshold, thenthe one of the plurality of probability values for the globalabnormality label should not exceed the detection probability threshold;and wherein a fourth one of the set of the plurality of desired labeldependency rules indicates that if the one of the plurality ofprobability values for the global abnormality label does not exceed thedetection probability threshold, then none of the set of the pluralityof probability values for the set of abnormality classification labelsshould exceed the detection probability threshold.
 15. A method forexecution by a model-assisted annotating system, the method comprising:receiving different ones of a first set of annotation data, wherein thefirst set of annotation data corresponds to a broad type of annotationdata output, wherein each of the first set of annotation data isgenerated by a corresponding one of a set of client devices based on abroad prompt presented via the corresponding one of the set of clientdevices to provide annotation data for one of a first set of medicalscans displayed by the corresponding one of the set of client devices;performing a first training step to train a computer vision modeloperable to generate the broad type of annotation data output byutilizing image data of the first set of medical scans and the first setof annotation data; generating a second set of annotation datacorresponding to the broad type of annotation data output by performingan inference function on image data of a second set of medical scans,wherein the inference function utilizes the computer vision model, andwherein each of the second set of annotation data is generated as outputof performing the inference function on a corresponding one of thesecond set of medical scans; receiving different ones of a set ofadditional annotation data, wherein the set of additional annotationdata corresponds to a specified type of annotation data output thatfurther specifies the broad type of annotation data output, wherein eachof the set of additional annotation data is generated by a correspondingone of the set of client devices based on a specified prompt presentedvia the corresponding one of the set of client devices to provideadditional annotations to further specify the corresponding one of thesecond set of annotation data displayed in conjunction with acorresponding one of the second set of medical scans by thecorresponding one of the set of client devices; performing a secondtraining step to generate an updated computer vision model operable togenerate the specified type of annotation data output by utilizing theset of additional annotation data to update the computer vision model;generating a third set of annotation data corresponding to the specifiedtype of annotation data output by performing an updated inferencefunction on image data of a third set of medical scans, wherein theupdated inference function utilizes the updated computer vision model,and wherein each of the third set of annotation data is generated asoutput of performing the updated inference function on a correspondingone of the third set of medical scans; and transmitting the third set ofmedical scans and the third set of annotation data for display.
 16. Themethod claim 15, wherein the set of additional annotation data includesa set of corrected annotation data, wherein each of the set of correctedannotation data is generated by one of the set of client devices inresponse to a prompt to provide corrections to the corresponding one ofthe second set of annotation data displayed in conjunction with one ofthe second set of medical scans, and wherein the second training steputilizes image data of the second set of medical scans as input labelsand utilizes the set of corrected annotation data as output labels. 17.The method of claim 15, wherein the broad type of annotation data outputcorresponds to global binary abnormality data, wherein performing thefirst training step includes utilizing image data as input labels andfurther includes utilizing a global binary abnormality label for each ofthe first set of annotation data as output labels, wherein the secondset of annotation data includes a global binary abnormality label foreach of the second set of medical scans, wherein the specified type ofannotation data output corresponds to abnormality characterization data,wherein the set of additional annotation data includes a set ofabnormality characterization data, wherein each of the set ofabnormality characterization data is generated by one of the set ofclient devices in response to the specified prompt to characterize anabnormality indicated in the corresponding one of the second set ofannotation data displayed in conjunction with one of the second set ofmedical scans, and wherein the second training step utilizes image dataof the second set of medical scans as input labels and utilizes the setof abnormality characterization data as output labels.
 18. The method ofclaim 15, wherein the second set of annotation data includes region ofinterest data generated as output of the inference function, and whereinthe region of interest data of one of the second set of annotation datais utilized to indicate a location of an abnormality indicated in theone of the second set of annotation data in the display of acorresponding one of the second set of medical scans.
 19. The method ofclaim 15, further comprising: generating inference trend data byautomatically evaluating the third set of annotation data to identify atleast one trend; and performing a third training step to generate asecond updated computer vision model based on the inference trend data.20. A method, comprising: receiving different ones of a first set ofannotation data, wherein the first set of annotation data corresponds toa broad type of annotation data output, wherein each of the first set ofannotation data is generated by a corresponding one of a set of clientdevices based on a broad prompt to provide annotation data for one of afirst set of medical scans displayed by the corresponding one of the setof client devices; performing a first training step to train a computervision model operable to generate the broad type of annotation dataoutput by utilizing image data of the first set of medical scans and thefirst set of annotation data; generating a second set of annotation datacorresponding to the broad type of annotation data output by performingan inference function on image data of a second set of medical scans,wherein the inference function utilizes the computer vision model, andwherein each of the second set of annotation data is generated as outputof performing the inference function on a corresponding one of thesecond set of medical scans; receiving different ones of a set ofadditional annotation data, wherein the set of additional annotationdata corresponds to a specified type of annotation data output thatfurther specifies the broad type of annotation data output, wherein eachof the set of additional annotation data is generated by a correspondingone of the set of client devices based on a specified prompt presentedvia the corresponding one of the set of client devices to provideadditional annotations to further specify the corresponding one of thesecond set of annotation data displayed in conjunction with acorresponding one of the second set of medical scans by thecorresponding one of the set of client devices; performing a secondtraining step to generate an updated computer vision model operable togenerate the specified type of annotation data output by utilizing theset of additional annotation data to update the computer vision model;generating a third set of annotation data corresponding to the specifiedtype of annotation data output by performing an updated inferencefunction on image data of a third set of medical scans, wherein theupdated inference function utilizes the updated computer vision model,and wherein each of the third set of annotation data is generated asoutput of performing the updated inference function on a correspondingone of the third set of medical scans; and sending the third set ofmedical scans and the third set of annotation data to a medical scandatabase for storage.